Skip to content

Commit def4c28

Browse files
tglsfdctglsfdc
committed
Change JSONB's on-disk format for improved performance.
The original design used an array of offsets into the variable-length portion of a JSONB container. However, such an array is basically uncompressible by simple compression techniques such as TOAST's LZ compressor. That's bad enough, but because the offset array is at the front, it tended to trigger the give-up-after-1KB heuristic in the TOAST code, so that the entire JSONB object was stored uncompressed; which was the root cause of bug #11109 from Larry White. To fix without losing the ability to extract a random array element in O(1) time, change this scheme so that most of the JEntry array elements hold lengths rather than offsets. With data that's compressible at all, there tend to be fewer distinct element lengths, so that there is scope for compression of the JEntry array. Every N'th entry is still an offset. To determine the length or offset of any specific element, we might have to examine up to N preceding JEntrys, but that's still O(1) so far as the total container size is concerned. Testing shows that this cost is negligible compared to other costs of accessing a JSONB field, and that the method does largely fix the incompressible-data problem. While at it, rearrange the order of elements in a JSONB object so that it's "all the keys, then all the values" not alternating keys and values. This doesn't really make much difference right at the moment, but it will allow providing a fast path for extracting individual object fields from large JSONB values stored EXTERNAL (ie, uncompressed), analogously to the existing optimization for substring extraction from large EXTERNAL text values. Bump catversion to denote the incompatibility in on-disk format. We will need to fix pg_upgrade to disallow upgrading jsonb data stored with 9.4 betas 1 and 2. Heikki Linnakangas and Tom Lane
1 parent ff27fcf commit def4c28

File tree

4 files changed

+357
-157
lines changed

4 files changed

+357
-157
lines changed

‎src/backend/utils/adt/jsonb.c

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -196,12 +196,12 @@ jsonb_from_cstring(char *json, int len)
196196
static size_t
197197
checkStringLen(size_t len)
198198
{
199-
if (len > JENTRY_POSMASK)
199+
if (len > JENTRY_OFFLENMASK)
200200
ereport(ERROR,
201201
(errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED),
202202
errmsg("string too long to represent as jsonb string"),
203203
errdetail("Due to an implementation restriction, jsonb strings cannot exceed %d bytes.",
204-
JENTRY_POSMASK)));
204+
JENTRY_OFFLENMASK)));
205205

206206
return len;
207207
}

0 commit comments

Comments
 (0)