Prevent infinite insertion loops in spgdoinsert().

tglsfdc · tglsfdc · commit c3c35a733c77 · 2021-05-14T15:07:34.000-04:00
Formerly we just relied on operator classes that assert longValuesOK to eventually shorten the leaf value enough to fit on an index page. That fails since the introduction of INCLUDE-column support (commit 09c1c6a), because the INCLUDE columns might alone take up more than a page, meaning no amount of leaf-datum compaction will get the job done. At least with spgtextproc.c, that leads to an infinite loop, since spgtextproc.c won't throw an error for not being able to shorten the leaf datum anymore. To fix without breaking cases that would otherwise work, add logic to spgdoinsert() to verify that the leaf tuple size is decreasing after each "choose" step. Some opclasses might not decrease the size on every single cycle, and in any case, alignment roundoff of the tuple size could obscure small gains. Therefore, allow up to 10 cycles without additional savings before throwing an error. (Perhaps this number will need adjustment, but it seems quite generous right now.) As long as we've developed this logic, let's back-patch it. The back branches don't have INCLUDE columns to worry about, but this seems like a good defense against possible bugs in operator classes. We already know that an infinite loop here is pretty unpleasant, so having a defense seems to outweigh the risk of breaking things. (Note that spgtextproc.c is actually the only known opclass with longValuesOK support, so that this is all moot for known non-core opclasses anyway.) Per report from Dilip Kumar. Discussion: https://postgr.es/m/CAFiTN-uxP_soPhVG840tRMQTBmtA_f_Y8N51G7DKYYqDh7XN-A@mail.gmail.com
diff --git a/doc/src/sgml/spgist.sgml b/doc/src/sgml/spgist.sgml
@@ -978,6 +978,18 @@ LANGUAGE C STRICT;
    fails to do that, the <acronym>SP-GiST</acronym> core resorts to
    extraordinary measures described in <xref linkend="spgist-all-the-same"/>.
   </para>
+
+  <para>
+   When <structfield>longValuesOK</structfield> is true, it is expected
+   that successive levels of the <acronym>SP-GiST</acronym> tree will
+   absorb more and more information into the prefixes and node labels of
+   the inner tuples, making the required leaf datum smaller and smaller,
+   so that eventually it will fit on a page.
+   To prevent bugs in operator classes from causing infinite insertion
+   loops, the <acronym>SP-GiST</acronym> core will raise an error if the
+   leaf datum does not become any smaller within ten cycles
+   of <function>choose</function> method calls.
+  </para>
  </sect2>
 
  <sect2 id="spgist-null-labels">
diff --git a/src/backend/access/spgist/spgdoinsert.c b/src/backend/access/spgist/spgdoinsert.c
@@ -669,7 +669,8 @@ checkAllTheSame(spgPickSplitIn *in, spgPickSplitOut *out, bool tooBig,
  * will eventually terminate if lack of balance is the issue.  If the tuple
  * is too big, we assume that repeated picksplit operations will eventually
  * make it small enough by repeated prefix-stripping.  A broken opclass could
- * make this an infinite loop, though.
+ * make this an infinite loop, though, so spgdoinsert() checks that the
+ * leaf datums get smaller each time.
  */
 static bool
 doPickSplit(Relation index, SpGistState *state,
@@ -1918,6 +1919,8 @@ spgdoinsert(Relation index, SpGistState *state,
 	int			level = 0;
 	Datum		leafDatums[INDEX_MAX_KEYS];
 	int			leafSize;
+	int			bestLeafSize;
+	int			numNoProgressCycles = 0;
 	SPPageDesc	current,
 				parent;
 	FmgrInfo   *procinfo = NULL;
@@ -1988,16 +1991,18 @@ spgdoinsert(Relation index, SpGistState *state,
 
 	/*
 	 * If it isn't gonna fit, and the opclass can't reduce the datum size by
-	 * suffixing, bail out now rather than getting into an endless loop.
+	 * suffixing, bail out now rather than doing a lot of useless work.
 	 */
-	if (leafSize > SPGIST_PAGE_CAPACITY && !state->config.longValuesOK)
+	if (leafSize > SPGIST_PAGE_CAPACITY &&
+		(isnull || !state->config.longValuesOK))
 		ereport(ERROR,
 				(errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED),
 				 errmsg("index row size %zu exceeds maximum %zu for index \"%s\"",
 						leafSize - sizeof(ItemIdData),
 						SPGIST_PAGE_CAPACITY - sizeof(ItemIdData),
 						RelationGetRelationName(index)),
 				 errhint("Values larger than a buffer page cannot be indexed.")));
+	bestLeafSize = leafSize;
 
 	/* Initialize "current" to the appropriate root page */
 	current.blkno = isnull ? SPGIST_NULL_BLKNO : SPGIST_ROOT_BLKNO;
@@ -2226,19 +2231,59 @@ spgdoinsert(Relation index, SpGistState *state,
 						leafSize += sizeof(ItemIdData);
 					}
 
+					/*
+					 * Check new tuple size; fail if it can't fit, unless the
+					 * opclass says it can handle the situation by suffixing.
+					 *
+					 * However, the opclass can only shorten the leaf datum,
+					 * which may not be enough to ever make the tuple fit,
+					 * since INCLUDE columns might alone use more than a page.
+					 * Depending on the opclass' behavior, that could lead to
+					 * an infinite loop --- spgtextproc.c, for example, will
+					 * just repeatedly generate an empty-string leaf datum
+					 * once it runs out of data.  Actual bugs in opclasses
+					 * might cause infinite looping, too.  To detect such a
+					 * loop, check to see if we are making progress by
+					 * reducing the leafSize in each pass.  This is a bit
+					 * tricky though.  Because of alignment considerations,
+					 * the total tuple size might not decrease on every pass.
+					 * Also, there are edge cases where the choose method
+					 * might seem to not make progress for a cycle or two.
+					 * Somewhat arbitrarily, we allow up to 10 no-progress
+					 * iterations before failing.  (This limit should be more
+					 * than MAXALIGN, to accommodate opclasses that trim one
+					 * byte from the leaf datum per pass.)
+					 */
+					if (leafSize > SPGIST_PAGE_CAPACITY)
+					{
+						bool		ok = false;
+
+						if (state->config.longValuesOK && !isnull)
+						{
+							if (leafSize < bestLeafSize)
+							{
+								ok = true;
+								bestLeafSize = leafSize;
+								numNoProgressCycles = 0;
+							}
+							else if (++numNoProgressCycles < 10)
+								ok = true;
+						}
+						if (!ok)
+							ereport(ERROR,
+									(errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED),
+									 errmsg("index row size %zu exceeds maximum %zu for index \"%s\"",
+											leafSize - sizeof(ItemIdData),
+											SPGIST_PAGE_CAPACITY - sizeof(ItemIdData),
+											RelationGetRelationName(index)),
+									 errhint("Values larger than a buffer page cannot be indexed.")));
+					}
+
 					/*
 					 * Loop around and attempt to insert the new leafDatum at
 					 * "current" (which might reference an existing child
 					 * tuple, or might be invalid to force us to find a new
 					 * page for the tuple).
-					 *
-					 * Note: if the opclass sets longValuesOK, we rely on the
-					 * choose function to eventually shorten the leafDatum
-					 * enough to fit on a page.  We could add a test here to
-					 * complain if the datum doesn't get visibly shorter each
-					 * time, but that could get in the way of opclasses that
-					 * "simplify" datums in a way that doesn't necessarily
-					 * lead to physical shortening on every cycle.
 					 */
 					break;
 				case spgAddNode:
diff --git a/src/test/modules/spgist_name_ops/expected/spgist_name_ops.out b/src/test/modules/spgist_name_ops/expected/spgist_name_ops.out
@@ -61,6 +61,12 @@ select * from t
  binary_upgrade_set_next_toast_pg_class_oid           |  1 | binary_upgrade_set_next_toast_pg_class_oid
 (9 rows)
 
+-- Verify clean failure when INCLUDE'd columns result in overlength tuple
+-- The error message details are platform-dependent, so show only SQLSTATE
+\set VERBOSITY sqlstate
+insert into t values(repeat('xyzzy', 12), 42, repeat('xyzzy', 4000));
+ERROR:  54000
+\set VERBOSITY default
 drop index t_f1_f2_f3_idx;
 create index on t using spgist(f1 name_ops_old) include(f2, f3);
 \d+ t_f1_f2_f3_idx
@@ -100,3 +106,7 @@ select * from t
  binary_upgrade_set_next_toast_pg_class_oid           |  1 | binary_upgrade_set_next_toast_pg_class_oid
 (9 rows)
 
+\set VERBOSITY sqlstate
+insert into t values(repeat('xyzzy', 12), 42, repeat('xyzzy', 4000));
+ERROR:  54000
+\set VERBOSITY default
diff --git a/src/test/modules/spgist_name_ops/sql/spgist_name_ops.sql b/src/test/modules/spgist_name_ops/sql/spgist_name_ops.sql
@@ -27,6 +27,12 @@ select * from t
   where f1 > 'binary_upgrade_set_n' and f1 < 'binary_upgrade_set_p'
   order by 1;
 
+-- Verify clean failure when INCLUDE'd columns result in overlength tuple
+-- The error message details are platform-dependent, so show only SQLSTATE
+\set VERBOSITY sqlstate
+insert into t values(repeat('xyzzy', 12), 42, repeat('xyzzy', 4000));
+\set VERBOSITY default
+
 drop index t_f1_f2_f3_idx;
 
 create index on t using spgist(f1 name_ops_old) include(f2, f3);
@@ -39,3 +45,7 @@ select * from t
 select * from t
   where f1 > 'binary_upgrade_set_n' and f1 < 'binary_upgrade_set_p'
   order by 1;
+
+\set VERBOSITY sqlstate
+insert into t values(repeat('xyzzy', 12), 42, repeat('xyzzy', 4000));
+\set VERBOSITY default
