CVS log for pgsql/src/backend/optimizer/util/clauses.c

Up to [PostgreSQL CVS Repository] / pgsql / src / backend / optimizer / util

Request diff between arbitrary revisions - Display revisions graphically

Keyword substitution: kv
Default branch: MAIN

Fix planner to make a reasonable assumption about the amount of memory space
used by array_agg(), string_agg(), and similar aggregate functions that use
"internal" as their transition datatype.  The previous coding thought this
took *no* extra space, since "internal" is pass-by-value; but actually these
aggregates typically consume a great deal of space.  Per bug #5608 from
Itagaki Takahiro, and fix suggestion from Hitoshi Harada.

Back-patch to 8.4, where array_agg was introduced.

Fix planner to make a reasonable assumption about the amount of memory space
used by array_agg(), string_agg(), and similar aggregate functions that use
"internal" as their transition datatype.  The previous coding thought this
took *no* extra space, since "internal" is pass-by-value; but actually these
aggregates typically consume a great deal of space.  Per bug #5608 from
Itagaki Takahiro, and fix suggestion from Hitoshi Harada.

Back-patch to 8.4, where array_agg was introduced.

Fix planner to make a reasonable assumption about the amount of memory space
used by array_agg(), string_agg(), and similar aggregate functions that use
"internal" as their transition datatype.  The previous coding thought this
took *no* extra space, since "internal" is pass-by-value; but actually these
aggregates typically consume a great deal of space.  Per bug #5608 from
Itagaki Takahiro, and fix suggestion from Hitoshi Harada.

Back-patch to 8.4, where array_agg was introduced.

Modify error context callback functions to not assume that they can fetch
catalog entries via SearchSysCache and related operations.  Although, at the
time that these callbacks are called by elog.c, we have not officially aborted
the current transaction, it still seems rather risky to initiate any new
catalog fetches.  In all these cases the needed information is readily
available in the caller and so it's just a matter of a bit of extra notation
to pass it to the callback.

Per crash report from Dennis Koegel.  I've concluded that the real fix for
his problem is to clear the error context stack at entry to proc_exit, but
it still seems like a good idea to make the callbacks a bit less fragile
for other cases.

Backpatch to 8.4.  We could go further back, but the patch doesn't apply
cleanly.  In the absence of proof that this fixes something and isn't just
paranoia, I'm not going to expend the effort.

Modify error context callback functions to not assume that they can fetch
catalog entries via SearchSysCache and related operations.  Although, at the
time that these callbacks are called by elog.c, we have not officially aborted
the current transaction, it still seems rather risky to initiate any new
catalog fetches.  In all these cases the needed information is readily
available in the caller and so it's just a matter of a bit of extra notation
to pass it to the callback.

Per crash report from Dennis Koegel.  I've concluded that the real fix for
his problem is to clear the error context stack at entry to proc_exit, but
it still seems like a good idea to make the callbacks a bit less fragile
for other cases.

Backpatch to 8.4.  We could go further back, but the patch doesn't apply
cleanly.  In the absence of proof that this fixes something and isn't just
paranoia, I'm not going to expend the effort.

pgindent run for 9.0

Wrap calls to SearchSysCache and related functions using macros.

The purpose of this change is to eliminate the need for every caller
of SearchSysCache, SearchSysCacheCopy, SearchSysCacheExists,
GetSysCacheOid, and SearchSysCacheList to know the maximum number
of allowable keys for a syscache entry (currently 4).  This will
make it far easier to increase the maximum number of keys in a
future release should we choose to do so, and it makes the code
shorter, too.

Design and review by Tom Lane.

Fix thinko in my recent change to put an explicit argisrow field in NullTest:
when the planner splits apart a ROW(...) IS NULL test, the argisrow values
of the component tests have to be determined from the component field types,
not copied from the original NullTest (in which argisrow is surely true).

Update copyright for the year 2010.

Add an "argisrow" field to NullTest nodes, following a plan made way back in
8.2beta but never carried out.  This avoids repetitive tests of whether the
argument is of scalar or composite type.  Also, be a bit more paranoid about
composite arguments in some places where we previously weren't checking.

Support ORDER BY within aggregate function calls, at long last providing a
non-kluge method for controlling the order in which values are fed to an
aggregate function.  At the same time eliminate the old implementation
restriction that DISTINCT was only supported for single-argument aggregates.

Possibly release-notable behavioral change: formerly, agg(DISTINCT x)
dropped null values of x unconditionally.  Now, it does so only if the
agg transition function is strict; otherwise nulls are treated as DISTINCT
normally would, ie, you get one copy.

Andrew Gierth, reviewed by Hitoshi Harada

Fix a bug introduced when set-returning SQL functions were made inline-able:
we have to cope with the possibility that the declared result rowtype contains
dropped columns.  This fails in 8.4, as per bug #5240.

While at it, be more paranoid about inserting binary coercions when inlining.
The pre-8.4 code did not really need to worry about that because it could not
inline at all in any case where an added coercion could change the behavior
of the function's statement.  However, when inlining a SRF we allow sorting,
grouping, and set-ops such as UNION.  In these cases, modifying one of the
targetlist entries that the sort/group/setop depends on could conceivably
change the behavior of the function's statement --- so don't inline when
such a case applies.

Fix a bug introduced when set-returning SQL functions were made inline-able:
we have to cope with the possibility that the declared result rowtype contains
dropped columns.  This fails in 8.4, as per bug #5240.

While at it, be more paranoid about inserting binary coercions when inlining.
The pre-8.4 code did not really need to worry about that because it could not
inline at all in any case where an added coercion could change the behavior
of the function's statement.  However, when inlining a SRF we allow sorting,
grouping, and set-ops such as UNION.  In these cases, modifying one of the
targetlist entries that the sort/group/setop depends on could conceivably
change the behavior of the function's statement --- so don't inline when
such a case applies.

Support use of function argument names to identify which actual arguments
match which function parameters.  The syntax uses AS, for example
	funcname(value AS arg1, anothervalue AS arg2)

Pavel Stehule

Teach simplify_boolean_equality to simplify the forms foo <> true and
foo <> false, along with its previous duties of simplifying foo = true
and foo = false.  (All of these are equivalent to just foo or NOT foo
as the case may be.)  It's not clear how often this is really useful;
but it costs almost nothing to do, and it seems some people think we
should be smart about such cases.  Per recent bug report.

8.4 pgindent run, with new combined Linux/FreeBSD/MinGW typedef list
provided by Andrew.

Get rid of the rather fuzzily defined FlattenedSubLink node type in favor of
making pull_up_sublinks() construct a full-blown JoinExpr tree representation
of IN/EXISTS SubLinks that it is able to convert to semi or anti joins.
This makes pull_up_sublinks() a shade more complex, but the gain in semantic
clarity is worth it.  I still have more to do in this area to address the
previously-discussed problems, but this commit in itself fixes at least one
bug in HEAD, as shown by added regression test case.

Arrange for function default arguments to be processed properly in expressions
that are set up for execution with ExecPrepareExpr rather than going through
the full planner process.  By introducing an explicit notion of "expression
planning", this patch also lays a bit of groundwork for maybe someday
allowing sub-selects in standalone expressions.

Fix an oversight in the function-default-arguments patch: after adding some
default expressions to a function call, eval_const_expressions must recurse on
those expressions.  Else they don't get simplified, and in particular we fail
to insert additional default arguments if any functions needing defaults are
in there.  Per report from Rushabh Lathia.

Update copyright for 2009.

Support window functions a la SQL:2008.

Hitoshi Harada, with some kibitzing from Heikki and Tom.

Code review for function default parameters patch.  Fix numerous problems as
per recent discussions.  In passing this also fixes a couple of bugs in
the previous variadic-parameters patch.

Add a concept of "placeholder" variables to the planner.  These are variables
that represent some expression that we desire to compute below the top level
of the plan, and then let that value "bubble up" as though it were a plain
Var (ie, a column value).

The immediate application is to allow sub-selects to be flattened even when
they are below an outer join and have non-nullable output expressions.
Formerly we couldn't flatten because such an expression wouldn't properly
go to NULL when evaluated above the outer join.  Now, we wrap it in a
PlaceHolderVar and arrange for the actual evaluation to occur below the outer
join.  When the resulting Var bubbles up through the join, it will be set to
NULL if necessary, yielding the correct results.  This fixes a planner
limitation that's existed since 7.1.

In future we might want to use this mechanism to re-introduce some form of
Hellerstein's "expensive functions" optimization, ie place the evaluation of
an expensive function at the most suitable point in the plan tree.

Improve the recently-added code for inlining set-returning functions so that
it can handle functions returning setof record.  The case was left undone
originally, but it turns out to be simple to fix.

Implement SQL-standard WITH clauses, including WITH RECURSIVE.

There are some unimplemented aspects: recursive queries must use UNION ALL
(should allow UNION too), and we don't have SEARCH or CYCLE clauses.
These might or might not get done for 8.4, but even without them it's a
pretty useful feature.

There are also a couple of small loose ends and definitional quibbles,
which I'll send a memo about to pgsql-hackers shortly.  But let's land
the patch now so we can get on with other development.

Yoshiyuki Asaba, with lots of help from Tatsuo Ishii and Tom Lane

Improve the plan cache invalidation mechanism to make it invalidate plans
when user-defined functions used in a plan are modified.  Also invalidate
plans when schemas, operators, or operator classes are modified; but for these
cases we just invalidate everything rather than tracking exact dependencies,
since these types of objects seldom change in a production database.

Tom Lane; loosely based on a patch by Martin Pihlak.

Extend the parser location infrastructure to include a location field in
most node types used in expression trees (both before and after parse
analysis).  This allows us to place an error cursor in many situations
where we formerly could not, because the information wasn't available
beyond the very first level of parse analysis.  There's a fair amount
of work still to be done to persuade individual ereport() calls to actually
include an error location, but this gets the initdb-forcing part of the
work out of the way; and the situation is already markedly better than
before for complaints about unimplementable implicit casts, such as
CASE and UNION constructs with incompatible alternative data types.
Per my proposal of a few days ago.

Teach eval_const_expressions() to simplify an ArrayCoerceExpr to a constant
when its input is constant and the element coercion function is immutable
(or nonexistent, ie, binary-coercible case).  This is an oversight in the
8.3 implementation of ArrayCoerceExpr, and its result is that certain cases
involving IN or NOT IN with constants don't get optimized as they should be.
Per experimentation with an example from Ow Mun Heng.

Teach eval_const_expressions() to simplify an ArrayCoerceExpr to a constant
when its input is constant and the element coercion function is immutable
(or nonexistent, ie, binary-coercible case).  This is an oversight in the
8.3 implementation of ArrayCoerceExpr, and its result is that certain cases
involving IN or NOT IN with constants don't get optimized as they should be.
Per experimentation with an example from Ow Mun Heng.

Move exprType(), exprTypmod(), expression_tree_walker(), and related routines
into nodes/nodeFuncs, so as to reduce wanton cross-subsystem #includes inside
the backend.  There's probably more that should be done along this line,
but this is a start anyway.

Arrange to convert EXISTS subqueries that are equivalent to hashable IN
subqueries into the same thing you'd have gotten from IN (except always with
unknownEqFalse = true, so as to get the proper semantics for an EXISTS).
I believe this fixes the last case within CVS HEAD in which an EXISTS could
give worse performance than an equivalent IN subquery.

The tricky part of this is that if the upper query probes the EXISTS for only
a few rows, the hashing implementation can actually be worse than the default,
and therefore we need to make a cost-based decision about which way to use.
But at the time when the planner generates plans for subqueries, it doesn't
really know how many times the subquery will be executed.  The least invasive
solution seems to be to generate both plans and postpone the choice until
execution.  Therefore, in a query that has been optimized this way, EXPLAIN
will show two subplans for the EXISTS, of which only one will actually get
executed.

There is a lot more that could be done based on this infrastructure: in
particular it's interesting to consider switching to the hash plan if we start
out using the non-hashed plan but find a lot more upper rows going by than we
expected.  I have therefore left some minor inefficiencies in place, such as
initializing both subplans even though we will currently only use one.

Implement SEMI and ANTI joins in the planner and executor.  (Semijoins replace
the old JOIN_IN code, but antijoins are new functionality.)  Teach the planner
to convert appropriate EXISTS and NOT EXISTS subqueries into semi and anti
joins respectively.  Also, LEFT JOINs with suitable upper-level IS NULL
filters are recognized as being anti joins.  Unify the InClauseInfo and
OuterJoinInfo infrastructure into "SpecialJoinInfo".  With that change,
it becomes possible to associate a SpecialJoinInfo with every join attempt,
which permits some cleanup of join selectivity estimation.  That needs to be
taken much further than this patch does, but the next step is to change the
API for oprjoin selectivity functions, which seems like material for a
separate patch.  So for the moment the output size estimates for semi and
especially anti joins are quite bogus.

Teach the system how to use hashing for UNION.  (INTERSECT/EXCEPT will follow,
but seem like a separate patch since most of the remaining work is on the
executor side.)  I took the opportunity to push selection of the grouping
operators for set operations into the parser where it belongs.  Otherwise this
is just a small exercise in making prepunion.c consider both alternatives.

As with the recent DISTINCT patch, this means we can UNION on datatypes that
can hash but not sort, and it means that UNION without ORDER BY is no longer
certain to produce sorted output.

Rearrange the querytree representation of ORDER BY/GROUP BY/DISTINCT items
as per my recent proposal:

1. Fold SortClause and GroupClause into a single node type SortGroupClause.
We were already relying on them to be struct-equivalent, so using two node
tags wasn't accomplishing much except to get in the way of comparing items
with equal().

2. Add an "eqop" field to SortGroupClause to carry the associated equality
operator.  This is cheap for the parser to get at the same time it's looking
up the sort operator, and storing it eliminates the need for repeated
not-so-cheap lookups during planning.  In future this will also let us
represent GROUP/DISTINCT operations on datatypes that have hash opclasses
but no btree opclasses (ie, they have equality but no natural sort order).
The previous representation simply didn't work for that, since its only
indicator of comparison semantics was a sort operator.

3. Add a hasDistinctOn boolean to struct Query to explicitly record whether
the distinctClause came from DISTINCT or DISTINCT ON.  This allows removing
some complicated and not 100% bulletproof code that attempted to figure
that out from the distinctClause alone.

This patch doesn't in itself create any new capability, but it's necessary
infrastructure for future attempts to use hash-based grouping for DISTINCT
and UNION/INTERSECT/EXCEPT.

Add code to eval_const_expressions() to support const-simplification of
CoerceViaIO nodes.  This improves the ability of the planner to deal with
cases where the node input is a constant.  Per bug #4170.

Add code to eval_const_expressions() to support const-simplification of
CoerceViaIO nodes.  This improves the ability of the planner to deal with
cases where the node input is a constant.  Per bug #4170.

Restructure some header files a bit, in particular heapam.h, by removing some
unnecessary #include lines in it.  Also, move some tuple routine prototypes and
macros to htup.h, which allows removal of heapam.h inclusion from some .c
files.

For this to work, a new header file access/sysattr.h needed to be created,
initially containing attribute numbers of system columns, for pg_dump usage.

While at it, make contrib ltree, intarray and hstore header files more
consistent with our header style.

Fix an oversight I made in a cleanup patch over a year ago:
eval_const_expressions needs to be passed the PlannerInfo ("root") structure,
because in some cases we want it to substitute values for Param nodes.
(So "constant" is not so constant as all that ...)  This mistake partially
disabled optimization of unnamed extended-Query statements in 8.3: in
particular the LIKE-to-indexscan optimization would never be applied if the
LIKE pattern was passed as a parameter, and constraint exclusion depending
on a parameter value didn't work either.

Fix an oversight I made in a cleanup patch over a year ago:
eval_const_expressions needs to be passed the PlannerInfo ("root") structure,
because in some cases we want it to substitute values for Param nodes.
(So "constant" is not so constant as all that ...)  This mistake partially
disabled optimization of unnamed extended-Query statements in 8.3: in
particular the LIKE-to-indexscan optimization would never be applied if the
LIKE pattern was passed as a parameter, and constraint exclusion depending
on a parameter value didn't work either.

Simplify and standardize conversions between TEXT datums and ordinary C
strings.  This patch introduces four support functions cstring_to_text,
cstring_to_text_with_len, text_to_cstring, and text_to_cstring_buffer, and
two macros CStringGetTextDatum and TextDatumGetCString.  A number of
existing macros that provided variants on these themes were removed.

Most of the places that need to make such conversions now require just one
function or macro call, in place of the multiple notational layers that used
to be needed.  There are no longer any direct calls of textout or textin,
and we got most of the places that were using handmade conversions via
memcpy (there may be a few still lurking, though).

This commit doesn't make any serious effort to eliminate transient memory
leaks caused by detoasting toasted text objects before they reach
text_to_cstring.  We changed PG_GETARG_TEXT_P to PG_GETARG_TEXT_PP in a few
places where it was easy, but much more could be done.

Brendan Jurd and Tom Lane

Arrange to "inline" SQL functions that appear in a query's FROM clause,
are declared to return set, and consist of just a single SELECT.  We
can replace the FROM-item with a sub-SELECT and then optimize much as
if we were dealing with a view.  Patch from Richard Rowell, cleaned up
by me.

The original implementation of polymorphic aggregates didn't really get the
checking of argument compatibility right; although the problem is only exposed
with multiple-input aggregates in which some arguments are polymorphic and
some are not.  Per bug #3852 from Sokolov Yura.

Update copyrights in source tree to 2008.

Save one syscache lookup when examining volatility or strictness of
OpExpr and related nodes.  We're going to have to set the opfuncid of
such nodes eventually (if we haven't already), so we might as well
exploit the opportunity to cache the function OID.  Buys back some
of the extra planner overhead noted by Guillaume Smet, though I still
need to fool with equivclass.c to really respond to that.

pgindent run for 8.3.

Ensure that the result of evaluating a function during constant-expression
simplification gets detoasted before it is incorporated into a Const node.
Otherwise, if an immutable function were to return a TOAST pointer (an
unlikely case, but it can be made to happen), we would end up with a plan
that depends on the continued existence of the out-of-line toast datum.

Ensure that the result of evaluating a function during constant-expression
simplification gets detoasted before it is incorporated into a Const node.
Otherwise, if an immutable function were to return a TOAST pointer (an
unlikely case, but it can be made to happen), we would end up with a plan
that depends on the continued existence of the out-of-line toast datum.

Ensure that the result of evaluating a function during constant-expression
simplification gets detoasted before it is incorporated into a Const node.
Otherwise, if an immutable function were to return a TOAST pointer (an
unlikely case, but it can be made to happen), we would end up with a plan
that depends on the continued existence of the out-of-line toast datum.

Ensure that the result of evaluating a function during constant-expression
simplification gets detoasted before it is incorporated into a Const node.
Otherwise, if an immutable function were to return a TOAST pointer (an
unlikely case, but it can be made to happen), we would end up with a plan
that depends on the continued existence of the out-of-line toast datum.

Ensure that the result of evaluating a function during constant-expression
simplification gets detoasted before it is incorporated into a Const node.
Otherwise, if an immutable function were to return a TOAST pointer (an
unlikely case, but it can be made to happen), we would end up with a plan
that depends on the continued existence of the out-of-line toast datum.

Ensure that the result of evaluating a function during constant-expression
simplification gets detoasted before it is incorporated into a Const node.
Otherwise, if an immutable function were to return a TOAST pointer (an
unlikely case, but it can be made to happen), we would end up with a plan
that depends on the continued existence of the out-of-line toast datum.

Make eval_const_expressions() preserve typmod when simplifying something like
null::char(3) to a simple Const node.  (It already worked for non-null values,
but not when we skipped evaluation of a strict coercion function.)  This
prevents loss of typmod knowledge in situations such as exhibited in bug
#3598.  Unfortunately there seems no good way to fix that bug in 8.1 and 8.2,
because they simply don't carry a typmod for a plain Const node.

In passing I made all the other callers of makeNullConst supply "real" typmod
values too, though I think it probably doesn't matter anywhere else.

Implement function-local GUC parameter settings, as per recent discussion.
There are still some loose ends: I didn't do anything about the SET FROM
CURRENT idea yet, and it's not real clear whether we are happy with the
interaction of SET LOCAL with function-local settings.  The documentation
is a bit spartan, too.

Separate parse-analysis for utility commands out of parser/analyze.c
(which now deals only in optimizable statements), and put that code
into a new file parser/parse_utilcmd.c.  This helps clarify and enforce
the design rule that utility statements shouldn't be processed during
the regular parse analysis phase; all interpretation of their meaning
should happen after they are given to ProcessUtility to execute.
(We need this because we don't retain any locks for a utility statement
that's in a plan cache, nor have any way to detect that it's stale.)

We are also able to simplify the API for parse_analyze() and related
routines, because they will now always return exactly one Query structure.

In passing, fix bug #3403 concerning trying to add a serial column to
an existing temp table (this is largely Heikki's work, but we needed
all that restructuring to make it safe).

Support UPDATE/DELETE WHERE CURRENT OF cursor_name, per SQL standard.

Along the way, allow FOR UPDATE in non-WITH-HOLD cursors; there may once
have been a reason to disallow that, but it seems to work now, and it's
really rather necessary if you want to select a row via a cursor and then
update it in a concurrent-safe fashion.

Original patch by Arul Shaji, rather heavily editorialized by Tom Lane.

Downgrade implicit casts to text to be assignment-only, except for the ones
from the other string-category types; this eliminates a lot of surprising
interpretations that the parser could formerly make when there was no directly
applicable operator.

Create a general mechanism that supports casts to and from the standard string
types (text,varchar,bpchar) for *every* datatype, by invoking the datatype's
I/O functions.  These new casts are assignment-only in the to-string direction,
explicit-only in the other, and therefore should create no surprising behavior.
Remove a bunch of thereby-obsoleted datatype-specific casting functions.

The "general mechanism" is a new expression node type CoerceViaIO that can
actually convert between *any* two datatypes if their external text
representations are compatible.  This is more general than needed for the
immediate feature, but might be useful in plpgsql or other places in future.

This commit does nothing about the issue that applying the concatenation
operator || to non-text types will now fail, often with strange error messages
due to misinterpreting the operator as array concatenation.  Since it often
(not always) worked before, we should either make it succeed or at least give
a more user-friendly error; but details are still under debate.

Peter Eisentraut and Tom Lane

Fix a thinko in my patch of a couple months ago for bug #3116: it did the
wrong thing when inlining polymorphic SQL functions, because it was using the
function's declared return type where it should have used the actual result
type of the current call.  In 8.1 and 8.2 this causes obvious failures even if
you don't have assertions turned on; in 8.0 and 7.4 it would only be a problem
if the inlined expression were used as an input to a function that did
run-time type determination on its inputs.  Add a regression test, since this
is evidently an under-tested area.

Fix a thinko in my patch of a couple months ago for bug #3116: it did the
wrong thing when inlining polymorphic SQL functions, because it was using the
function's declared return type where it should have used the actual result
type of the current call.  In 8.1 and 8.2 this causes obvious failures even if
you don't have assertions turned on; in 8.0 and 7.4 it would only be a problem
if the inlined expression were used as an input to a function that did
run-time type determination on its inputs.  Add a regression test, since this
is evidently an under-tested area.

Fix a thinko in my patch of a couple months ago for bug #3116: it did the
wrong thing when inlining polymorphic SQL functions, because it was using the
function's declared return type where it should have used the actual result
type of the current call.  In 8.1 and 8.2 this causes obvious failures even if
you don't have assertions turned on; in 8.0 and 7.4 it would only be a problem
if the inlined expression were used as an input to a function that did
run-time type determination on its inputs.  Add a regression test, since this
is evidently an under-tested area.

Fix a thinko in my patch of a couple months ago for bug #3116: it did the
wrong thing when inlining polymorphic SQL functions, because it was using the
function's declared return type where it should have used the actual result
type of the current call.  In 8.1 and 8.2 this causes obvious failures even if
you don't have assertions turned on; in 8.0 and 7.4 it would only be a problem
if the inlined expression were used as an input to a function that did
run-time type determination on its inputs.  Add a regression test, since this
is evidently an under-tested area.

Fix a thinko in my patch of a couple months ago for bug #3116: it did the
wrong thing when inlining polymorphic SQL functions, because it was using the
function's declared return type where it should have used the actual result
type of the current call.  In 8.1 and 8.2 this causes obvious failures even if
you don't have assertions turned on; in 8.0 and 7.4 it would only be a problem
if the inlined expression were used as an input to a function that did
run-time type determination on its inputs.  Add a regression test, since this
is evidently an under-tested area.

Marginal performance hack: avoid unnecessary work in expression_tree_mutator.
We can just palloc, instead of using makeNode, when we are going to
overwrite the whole node anyway in the FLATCOPY macro.  Also, use
FLATCOPY instead of copyObject for common node types Var and Const.

Modify processing of DECLARE CURSOR and EXPLAIN so that they can resolve the
types of unspecified parameters when submitted via extended query protocol.
This worked in 8.2 but I had broken it during plancache changes.  DECLARE
CURSOR is now treated almost exactly like a plain SELECT through parse
analysis, rewrite, and planning; only just before sending to the executor
do we divert it away to ProcessUtility.  This requires a special-case check
in a number of places, but practically all of them were already special-casing
SELECT INTO, so it's not too ugly.  (Maybe it would be a good idea to merge
the two by treating IntoClause as a form of utility statement?  Not going to
worry about that now, though.)  That approach doesn't work for EXPLAIN,
however, so for that I punted and used a klugy solution of running parse
analysis an extra time if under extended query protocol.

Support enum data types.  Along the way, use macros for the values of
pg_type.typtype whereever practical.  Tom Dunstan, with some kibitzing
from Tom Lane.

Fix array coercion expressions to ensure that the correct volatility is
seen by code inspecting the expression.  The best way to do this seems
to be to drop the original representation as a function invocation, and
instead make a special expression node type that represents applying
the element-type coercion function to each array element.  In this way
the element function is exposed and will be checked for volatility.
Per report from Guillaume Smet.

Fix up the remaining places where the expression node structure would lose
available information about the typmod of an expression; namely, Const,
ArrayRef, ArrayExpr, and EXPR and ARRAY SubLinks.  In the ArrayExpr and
SubLink cases it wasn't really the data structure's fault, but exprTypmod()
being lazy.  This seems like a good idea in view of the expected increase in
typmod usage from Teodor's work to allow user-defined types to have typmods.
In particular this responds to the concerns we had about eliminating the
special-purpose hack that exprTypmod() used to have for BPCHAR Consts.
We can now tell whether or not such a Const has been cast to a specific
length, and report or display properly if so.

initdb forced due to changes in stored rules.

First phase of plan-invalidation project: create a plan cache management
module and teach PREPARE and protocol-level prepared statements to use it.
In service of this, rearrange utility-statement processing so that parse
analysis does not assume table schemas can't change before execution for
utility statements (necessary because we don't attempt to re-acquire locks
for utility statements when reusing a stored plan).  This requires some
refactoring of the ProcessUtility API, but it ends up cleaner anyway,
for instance we can get rid of the QueryContext global.

Still to do: fix up SPI and related code to use the plan cache; I'm tempted to
try to make SQL functions use it too.  Also, there are at least some aspects
of system state that we want to ensure remain the same during a replan as in
the original processing; search_path certainly ought to behave that way for
instance, and perhaps there are others.

Fix oversight in original coding of inline_function(): since
check_sql_fn_retval allows binary-compatibility cases, the expression
extracted from an inline-able SQL function might have a type that is only
binary-compatible with the declared function result type.  To avoid possibly
changing the semantics of the expression, we should insert a RelabelType node
in such cases.  This has only been shown to have bad consequences in recent
8.1 and up releases, but I suspect there may be failure cases in the older
branches too, so patch it all the way back.  Per bug #3116 from Greg Mullane.

Along the way, fix an omission in eval_const_expressions_mutator: it failed
to copy the relabelformat field when processing a RelabelType.  No known
observable failures from this, but it definitely isn't intended behavior.

Fix oversight in original coding of inline_function(): since
check_sql_fn_retval allows binary-compatibility cases, the expression
extracted from an inline-able SQL function might have a type that is only
binary-compatible with the declared function result type.  To avoid possibly
changing the semantics of the expression, we should insert a RelabelType node
in such cases.  This has only been shown to have bad consequences in recent
8.1 and up releases, but I suspect there may be failure cases in the older
branches too, so patch it all the way back.  Per bug #3116 from Greg Mullane.

Along the way, fix an omission in eval_const_expressions_mutator: it failed
to copy the relabelformat field when processing a RelabelType.  No known
observable failures from this, but it definitely isn't intended behavior.

Fix oversight in original coding of inline_function(): since
check_sql_fn_retval allows binary-compatibility cases, the expression
extracted from an inline-able SQL function might have a type that is only
binary-compatible with the declared function result type.  To avoid possibly
changing the semantics of the expression, we should insert a RelabelType node
in such cases.  This has only been shown to have bad consequences in recent
8.1 and up releases, but I suspect there may be failure cases in the older
branches too, so patch it all the way back.  Per bug #3116 from Greg Mullane.

Along the way, fix an omission in eval_const_expressions_mutator: it failed
to copy the relabelformat field when processing a RelabelType.  No known
observable failures from this, but it definitely isn't intended behavior.

Fix oversight in original coding of inline_function(): since
check_sql_fn_retval allows binary-compatibility cases, the expression
extracted from an inline-able SQL function might have a type that is only
binary-compatible with the declared function result type.  To avoid possibly
changing the semantics of the expression, we should insert a RelabelType node
in such cases.  This has only been shown to have bad consequences in recent
8.1 and up releases, but I suspect there may be failure cases in the older
branches too, so patch it all the way back.  Per bug #3116 from Greg Mullane.

Along the way, fix an omission in eval_const_expressions_mutator: it failed
to copy the relabelformat field when processing a RelabelType.  No known
observable failures from this, but it definitely isn't intended behavior.

Fix oversight in original coding of inline_function(): since
check_sql_fn_retval allows binary-compatibility cases, the expression
extracted from an inline-able SQL function might have a type that is only
binary-compatible with the declared function result type.  To avoid possibly
changing the semantics of the expression, we should insert a RelabelType node
in such cases.  This has only been shown to have bad consequences in recent
8.1 and up releases, but I suspect there may be failure cases in the older
branches too, so patch it all the way back.  Per bug #3116 from Greg Mullane.

Along the way, fix an omission in eval_const_expressions_mutator: it failed
to copy the relabelformat field when processing a RelabelType.  No known
observable failures from this, but it definitely isn't intended behavior.

Turn the rangetable used by the executor into a flat list, and avoid storing
useless substructure for its RangeTblEntry nodes.  (I chose to keep using the
same struct node type and just zero out the link fields for unneeded info,
rather than making a separate ExecRangeTblEntry type --- it seemed too
fragile to have two different rangetable representations.)

Along the way, put subplans into a list in the toplevel PlannedStmt node,
and have SubPlan nodes refer to them by list index instead of direct pointers.
Vadim wanted to do that years ago, but I never understood what he was on about
until now.  It makes things a *whole* lot more robust, because we can stop
worrying about duplicate processing of subplans during expression tree
traversals.  That's been a constant source of bugs, and it's finally gone.

There are some consequent simplifications yet to be made, like not using
a separate EState for subplans in the executor, but I'll tackle that later.

Get rid of some old and crufty global variables in the planner.  When
this code was last gone over, there wasn't really any alternative to
globals because we didn't have the PlannerInfo struct being passed all
through the planner code.  Now that we do, we can restructure things
to avoid non-reentrancy.  I'm fooling with this because otherwise I'd
have had to add another global variable for the planned compact
range table list.

Teach find_nonnullable_rels to handle OR cases: if every arm of an OR
forces a particular relation nonnullable, then we can say that the OR does.
This is worth a little extra trouble since it may allow reduction of
outer joins to plain joins.

Repair insufficiently careful type checking for SQL-language functions:
we should check that the function code returns the claimed result datatype
every time we parse the function for execution.  Formerly, for simple
scalar result types we assumed the creation-time check was sufficient, but
this fails if the function selects from a table that's been redefined since
then, and even more obviously fails if check_function_bodies had been OFF.

This is a significant security hole: not only can one trivially crash the
backend, but with appropriate misuse of pass-by-reference datatypes it is
possible to read out arbitrary locations in the server process's memory,
which could allow retrieving database content the user should not be able
to see.  Our thanks to Jeff Trout for the initial report.

Security: CVE-2007-0555

Repair insufficiently careful type checking for SQL-language functions:
we should check that the function code returns the claimed result datatype
every time we parse the function for execution.  Formerly, for simple
scalar result types we assumed the creation-time check was sufficient, but
this fails if the function selects from a table that's been redefined since
then, and even more obviously fails if check_function_bodies had been OFF.

This is a significant security hole: not only can one trivially crash the
backend, but with appropriate misuse of pass-by-reference datatypes it is
possible to read out arbitrary locations in the server process's memory,
which could allow retrieving database content the user should not be able
to see.  Our thanks to Jeff Trout for the initial report.

Security: CVE-2007-0555

Repair insufficiently careful type checking for SQL-language functions:
we should check that the function code returns the claimed result datatype
every time we parse the function for execution.  Formerly, for simple
scalar result types we assumed the creation-time check was sufficient, but
this fails if the function selects from a table that's been redefined since
then, and even more obviously fails if check_function_bodies had been OFF.

This is a significant security hole: not only can one trivially crash the
backend, but with appropriate misuse of pass-by-reference datatypes it is
possible to read out arbitrary locations in the server process's memory,
which could allow retrieving database content the user should not be able
to see.  Our thanks to Jeff Trout for the initial report.

Security: CVE-2007-0555

Repair insufficiently careful type checking for SQL-language functions:
we should check that the function code returns the claimed result datatype
every time we parse the function for execution.  Formerly, for simple
scalar result types we assumed the creation-time check was sufficient, but
this fails if the function selects from a table that's been redefined since
then, and even more obviously fails if check_function_bodies had been OFF.

This is a significant security hole: not only can one trivially crash the
backend, but with appropriate misuse of pass-by-reference datatypes it is
possible to read out arbitrary locations in the server process's memory,
which could allow retrieving database content the user should not be able
to see.  Our thanks to Jeff Trout for the initial report.

Security: CVE-2007-0555

Repair insufficiently careful type checking for SQL-language functions:
we should check that the function code returns the claimed result datatype
every time we parse the function for execution.  Formerly, for simple
scalar result types we assumed the creation-time check was sufficient, but
this fails if the function selects from a table that's been redefined since
then, and even more obviously fails if check_function_bodies had been OFF.

This is a significant security hole: not only can one trivially crash the
backend, but with appropriate misuse of pass-by-reference datatypes it is
possible to read out arbitrary locations in the server process's memory,
which could allow retrieving database content the user should not be able
to see.  Our thanks to Jeff Trout for the initial report.

Security: CVE-2007-0555

Wording cleanup for error messages.  Also change can't -> cannot.

Standard English uses "may", "can", and "might" in different ways:

        may - permission, "You may borrow my rake."

        can - ability, "I can lift that log."

        might - possibility, "It might rain today."

Unfortunately, in conversational English, their use is often mixed, as
in, "You may use this variable to do X", when in fact, "can" is a better
choice.  Similarly, "It may crash" is better stated, "It might crash".

Add COST and ROWS options to CREATE/ALTER FUNCTION, plus underlying pg_proc
columns procost and prorows, to allow simple user adjustment of the estimated
cost of a function call, as well as control of the estimated number of rows
returned by a set-returning function.  We might eventually wish to extend this
to allow function-specific estimation routines, but there seems to be
consensus that we should try a simple constant estimate first.  In particular
this provides a relatively simple way to control the order in which different
WHERE clauses are applied in a plan node, which is a Good Thing in view of the
fact that the recent EquivalenceClass planner rewrite made that much less
predictable than before.

Add a note pointing out that is_pseudo_constant_clause() doesn't check
for aggregates.  This is OK for current uses but could burn somebody
someday...

Change the planner-to-executor API so that the planner tells the executor
which comparison operators to use for plan nodes involving tuple comparison
(Agg, Group, Unique, SetOp).  Formerly the executor looked up the default
equality operator for the datatype, which was really pretty shaky, since it's
possible that the data being fed to the node is sorted according to some
nondefault operator class that could have an incompatible idea of equality.
The planner knows what it has sorted by and therefore can provide the right
equality operator to use.  Also, this change moves a couple of catalog lookups
out of the executor and into the planner, which should help startup time for
pre-planned queries by some small amount.  Modify the planner to remove some
other cavalier assumptions about always being able to use the default
operators.  Also add "nulls first/last" info to the Plan node for a mergejoin
--- neither the executor nor the planner can cope yet, but at least the API is
in place.

Support ORDER BY ... NULLS FIRST/LAST, and add ASC/DESC/NULLS FIRST/NULLS LAST
per-column options for btree indexes.  The planner's support for this is still
pretty rudimentary; it does not yet know how to plan mergejoins with
nondefault ordering options.  The documentation is pretty rudimentary, too.
I'll work on improving that stuff later.

Note incompatible change from prior behavior: ORDER BY ... USING will now be
rejected if the operator is not a less-than or greater-than member of some
btree opclass.  This prevents less-than-sane behavior if an operator that
doesn't actually define a proper sort ordering is selected.

Update CVS HEAD for 2007 copyright.  Back branches are typically not
back-stamped for this.

Code review for XML patch.  Instill a bit of sanity in the location of
the XmlExpr code in various lists, use a representation that has some hope
of reverse-listing correctly (though it's still a de-escaping function
shy of correctness), generally try to make it look more like Postgres
coding conventions.

Restructure operator classes to allow improved handling of cross-data-type
cases.  Operator classes now exist within "operator families".  While most
families are equivalent to a single class, related classes can be grouped
into one family to represent the fact that they are semantically compatible.
Cross-type operators are now naturally adjunct parts of a family, without
having to wedge them into a particular opclass as we had done originally.

This commit restructures the catalogs and cleans up enough of the fallout so
that everything still works at least as well as before, but most of the work
needed to actually improve the planner's behavior will come later.  Also,
there are not yet CREATE/DROP/ALTER OPERATOR FAMILY commands; the only way
to create a new family right now is to allow CREATE OPERATOR CLASS to make
one by default.  I owe some more documentation work, too.  But that can all
be done in smaller pieces once this infrastructure is in place.

Initial SQL/XML support: xml data type and initial set of functions.

expression_tree_walker failed to let walker function see the immediate child
node of a SubLink or SubPlan testexpr field.  Bug resulted from replacing
the old lefthand/exprs list fields with a simple expression field, and not
remembering that expression_tree_walker is coded to save a few cycles by
recursing directly to self on list fields (on the assumption the walker
isn't interested in List nodes per se).  On non-list fields it must of
course call the walker.  Possibly that hack isn't worth the risk of more
such bugs, but I'll leave it be for now.  Per bug report from James Robinson.

pgindent run for 8.2.

Fix IS NULL and IS NOT NULL tests on row-valued expressions to conform to
the SQL spec, viz IS NULL is true if all the row's fields are null, IS NOT
NULL is true if all the row's fields are not null.  The former coding got
this right for a limited number of cases with IS NULL (ie, those where it
could disassemble a ROW constructor at parse time), but was entirely wrong
for IS NOT NULL.  Per report from Teodor.

I desisted from changing the behavior for arrays, since on closer inspection
it's not clear that there's any support for that in the SQL spec.  This
probably needs more consideration.

Change processing of extended-Query mode so that an unnamed statement
that has parameters is always planned afresh for each Bind command,
treating the parameter values as constants in the planner.  This removes
the performance penalty formerly often paid for using out-of-line
parameters --- with this definition, the planner can do constant folding,
LIKE optimization, etc.  After a suggestion by Andrew@supernews.

Tweak SPI_cursor_open to allow INSERT/UPDATE/DELETE RETURNING; this was
merely a matter of fixing the error check, since the underlying Portal
infrastructure already handles it.  This in turn allows these statements
to be used in some existing plpgsql and plperl contexts, such as a
plpgsql FOR loop.  Also, do some marginal code cleanup in places that
were being sloppy about distinguishing SELECT from SELECT INTO.

Add INSERT/UPDATE/DELETE RETURNING, with basic docs and regression tests.
plpgsql support to come later.  Along the way, convert execMain's
SELECT INTO support into a DestReceiver, in order to eliminate some ugly
special cases.

Jonah Harris and Tom Lane

Teach eval_const_expressions to simplify BooleanTest nodes that have
constant input.  Seems worth doing because rule rewriter inserts
IS NOT TRUE tests into WHERE clauses.

Add support for multi-row VALUES clauses as part of INSERT statements
(e.g. "INSERT ... VALUES (...), (...), ...") and elsewhere as allowed
by the spec. (e.g. similar to a FROM clause subselect). initdb required.
Joe Conway and Tom Lane.

Aggregate functions now support multiple input arguments.  I also took
the opportunity to treat COUNT(*) as a zero-argument aggregate instead
of the old hack that equated it to COUNT(1); this is materially cleaner
(no more weird ANYOID cases) and ought to be at least a tiny bit faster.
Original patch by Sergey Koposov; review, documentation, simple regression
tests, pg_dump and psql support by moi.

Remove 576 references of include files that were not needed.

Revise the planner's handling of "pseudoconstant" WHERE clauses, that is
clauses containing no variables and no volatile functions.  Such a clause
can be used as a one-time qual in a gating Result plan node, to suppress
plan execution entirely when it is false.  Even when the clause is true,
putting it in a gating node wins by avoiding repeated evaluation of the
clause.  In previous PG releases, query_planner() would do this for
pseudoconstant clauses appearing at the top level of the jointree, but
there was no ability to generate a gating Result deeper in the plan tree.
To fix it, get rid of the special case in query_planner(), and instead
process pseudoconstant clauses through the normal RestrictInfo qual
distribution mechanism.  When a pseudoconstant clause is found attached to
a path node in create_plan(), pull it out and generate a gating Result at
that point.  This requires special-casing pseudoconstants in selectivity
estimation and cost_qual_eval, but on the whole it's pretty clean.
It probably even makes the planner a bit faster than before for the normal
case of no pseudoconstants, since removing pull_constant_clauses saves one
useless traversal of the qual tree.  Per gripe from Phil Frost.

Fix problems with cached tuple descriptors disappearing while still in use
by creating a reference-count mechanism, similar to what we did a long time
ago for catcache entries.  The back branches have an ugly solution involving
lots of extra copies, but this way is more efficient.  Reference counting is
only applied to tupdescs that are actually in caches --- there seems no need
to use it for tupdescs that are generated in the executor, since they'll go
away during plan shutdown by virtue of being in the per-query memory context.
Neil Conway and Tom Lane

Simplify ParamListInfo data structure to support only numbered parameters,
not named ones, and replace linear searches of the list with array indexing.
The named-parameter support has been dead code for many years anyway,
and recent profiling suggests that the searching was costing a noticeable
amount of performance for complex queries.

Improve parser so that we can show an error cursor position for errors
during parse analysis, not only errors detected in the flex/bison stages.
This is per my earlier proposal.  This commit includes all the basic
infrastructure, but locations are only tracked and reported for errors
involving column references, function calls, and operators.  More could
be done later but this seems like a good set to start with.  I've also
moved the ReportSyntaxErrorPosition logic out of psql and into libpq,
which should make it available to more people --- even within psql this
is an improvement because warnings weren't handled by ReportSyntaxErrorPosition.

Update copyright for 2006.  Update scripts.

Improve the tests to see if ScalarArrayOpExpr is strict.  Original coding
would basically punt in all cases for 'foo <> ALL (array)', which resulted
in a performance regression for NOT IN compared to what we were doing in
8.1 and before.  Per report from Pavel Stehule.

Restructure planner's handling of inheritance.  Rather than processing
inheritance trees on-the-fly, which pretty well constrained us to considering
only one way of planning inheritance, expand inheritance sets during the
planner prep phase, and build a side data structure that can be consulted
later to find which RTEs are members of which inheritance sets.  As proof of
concept, use the data structure to plan joins against inheritance sets more
efficiently: we can now use indexes on the set members in inner-indexscan
joins.  (The generated plans could be improved further, but it'll take some
executor changes.)  This data structure will also support handling UNION ALL
subqueries in the same way as inheritance sets, but that aspect of it isn't
finished yet.

Allow row comparisons to be used as indexscan qualifications.
This completes the project to upgrade our handling of row comparisons.

Implement SQL-compliant treatment of row comparisons for < <= > >= cases
(previously we only did = and <> correctly).  Also, allow row comparisons
with any operators that are in btree opclasses, not only those with these
specific names.  This gets rid of a whole lot of indefensible assumptions
about the behavior of particular operators based on their names ... though
it's still true that IN and NOT IN expand to "= ANY".  The patch adds a
RowCompareExpr expression node type, and makes some changes in the
representation of ANY/ALL/ROWCOMPARE SubLinks so that they can share code
with RowCompareExpr.

I have not yet done anything about making RowCompareExpr an indexable
operator, but will look at that soon.

initdb forced due to changes in stored rules.

Teach planner how to rearrange join order for some classes of OUTER JOIN.
Per my recent proposal.  I ended up basing the implementation on the
existing mechanism for enforcing valid join orders of IN joins --- the
rules for valid outer-join orders are somewhat similar.

Re-run pgindent, fixing a problem where comment lines after a blank
comment line where output as too long, and update typedefs for /lib
directory.  Also fix case where identifiers were used as variable names
in the backend, but as typedefs in ecpg (favor the backend for
indenting).

Backpatch to 8.1.X.

Re-run pgindent, fixing a problem where comment lines after a blank
comment line where output as too long, and update typedefs for /lib
directory.  Also fix case where identifiers were used as variable names
in the backend, but as typedefs in ecpg (favor the backend for
indenting).

Backpatch to 8.1.X.

Make SQL arrays support null elements.  This commit fixes the core array
functionality, but I still need to make another pass looking at places
that incidentally use arrays (such as ACL manipulation) to make sure they
are null-safe.  Contrib needs work too.
I have not changed the behaviors that are still under discussion about
array comparison and what to do with lower bounds.

Standard pgindent run for 8.1.

Don't try to constant-fold functions returning RECORD.  We were never
able to do this before, but I had tried to make an exception for functions
with OUT parameters.  Michael Fuhr found one problem with it already, and
I found another, which was it didn't work for strict functions with a
NULL input.  While both of these could be worked around, the probability
that there are more gotchas seems high; I think prudence dictates just
reverting to the former behavior for now.  Accordingly, remove the kluge
added to get_expr_result_type() for Michael's case.

Add Oracle-compatible GREATEST and LEAST functions.  Pavel Stehule

Remove planner's private fields from Query struct, and put them into
a new PlannerInfo struct, which is passed around instead of the bare
Query in all the planning code.  This commit is essentially just a
code-beautification exercise, but it does open the door to making
larger changes to the planner data structures without having to muck
with the widely-known Query struct.

Teach the planner to remove SubqueryScan nodes from the plan if they
aren't doing anything useful (ie, neither selection nor projection).
Also, extend to SubqueryScan the hacks already in place to avoid
unnecessary ExecProject calls when the result would just be the same
tuple the subquery already delivered.  This saves some overhead in
UNION and other set operations, as well as avoiding overhead for
unflatten-able subqueries.  Per example from Sokolov Yura.

Turns out that my recent elimination of the 'redundant' flatten_andors()
code in prepqual.c had a small drawback: the flatten_andors code was
able to cope with deeply nested AND/OR structures (like 10000 ORs in
a row), whereas eval_const_expressions tends to recurse until it
overruns the stack.  Revise eval_const_expressions so that it doesn't
choke on deeply nested ANDs or ORs.

Don't try to constant-fold functions returning RECORD, since the optimizer
isn't presently set up to pass them an expected tuple descriptor.  Bug has
been there since 7.3 but was just recently reported by Thomas Hallgren.

Don't try to constant-fold functions returning RECORD, since the optimizer
isn't presently set up to pass them an expected tuple descriptor.  Bug has
been there since 7.3 but was just recently reported by Thomas Hallgren.

Don't try to constant-fold functions returning RECORD, since the optimizer
isn't presently set up to pass them an expected tuple descriptor.  Bug has
been there since 7.3 but was just recently reported by Thomas Hallgren.

Don't try to constant-fold functions returning RECORD, since the optimizer
isn't presently set up to pass them an expected tuple descriptor.  Bug has
been there since 7.3 but was just recently reported by Thomas Hallgren.

Make constant-folding produce sane output for COALESCE(NULL,NULL),
that is a plain NULL and not a COALESCE with no inputs.  Fixes crash
reported by Michael Williamson.

Make constant-folding produce sane output for COALESCE(NULL,NULL),
that is a plain NULL and not a COALESCE with no inputs.  Fixes crash
reported by Michael Williamson.

Make constant-folding produce sane output for COALESCE(NULL,NULL),
that is a plain NULL and not a COALESCE with no inputs.  Fixes crash
reported by Michael Williamson.

Merge Resdom nodes into TargetEntry nodes to simplify code and save a
few palloc's.  I also chose to eliminate the restype and restypmod fields
entirely, since they are redundant with information stored in the node's
contained expression; re-examining the expression at need seems simpler
and more reliable than trying to keep restype/restypmod up to date.

initdb forced due to change in contents of stored rules.

First phase of OUT-parameters project.  We can now define and use SQL
functions with OUT parameters.  The various PLs still need work, as does
pg_dump.  Rudimentary docs and regression tests included.

Convert oidvector and int2vector into variable-length arrays.  This
change saves a great deal of space in pg_proc and its primary index,
and it eliminates the former requirement that INDEX_MAX_KEYS and
FUNC_MAX_ARGS have the same value.  INDEX_MAX_KEYS is still embedded
in the on-disk representation (because it affects index tuple header
size), but FUNC_MAX_ARGS is not.  I believe it would now be possible
to increase FUNC_MAX_ARGS at little cost, but haven't experimented yet.
There are still a lot of vestigial references to FUNC_MAX_ARGS, which
I will clean up in a separate pass.  However, getting rid of it
altogether would require changing the FunctionCallInfoData struct,
and I'm not sure I want to buy into that.

Rethink the order of expression preprocessing: eval_const_expressions
really ought to run before canonicalize_qual, because it can now produce
forms that canonicalize_qual knows how to improve (eg, NOT clauses).
Also, because eval_const_expressions already knows about flattening
nested ANDs and ORs into N-argument form, the initial flatten_andors
pass in canonicalize_qual is now completely redundant and can be
removed.  This doesn't save a whole lot of code, but the time and
palloc traffic eliminated is a useful gain on large expression trees.

Teach const-expression simplification to simplify boolean equality cases,
that is 'x = true' becomes 'x' and 'x = false' becomes 'NOT x'.  This isn't
all that amazingly useful in itself, but it ensures that we will recognize
the different forms as being logically equivalent when checking partial
index predicates.  Per example from Patrick Clery.

Adjust constant-folding of CASE expressions so that the simple comparison
form of CASE (eg, CASE 0 WHEN 1 THEN ...) can be constant-folded as it
was in 7.4.  Also, avoid constant-folding result expressions that are
certainly unreachable --- the former coding was a bit cavalier about this
and could generate unexpected results for all-constant CASE expressions.
Add regression test cases.  Per report from Vlad Marchenko.

Adjust constant-folding of CASE expressions so that the simple comparison
form of CASE (eg, CASE 0 WHEN 1 THEN ...) can be constant-folded as it
was in 7.4.  Also, avoid constant-folding result expressions that are
certainly unreachable --- the former coding was a bit cavalier about this
and could generate unexpected results for all-constant CASE expressions.
Add regression test cases.  Per report from Vlad Marchenko.

Improve planner's estimation of the space needed for HashAgg plans:
look at the actual aggregate transition datatypes and the actual overhead
needed by nodeAgg.c, instead of using pessimistic round numbers.
Per a discussion with Michael Tiemann.

Improve planner's estimation of the space needed for HashAgg plans:
look at the actual aggregate transition datatypes and the actual overhead
needed by nodeAgg.c, instead of using pessimistic round numbers.
Per a discussion with Michael Tiemann.

Tag appropriate files for rc3

Also performed an initial run through of upgrading our Copyright date to
extend to 2005 ... first run here was very simple ... change everything
where: grep 1996-2004 && the word 'Copyright' ... scanned through the
generated list with 'less' first, and after, to make sure that I only
picked up the right entries ...

Instead of supposing (wrongly, in the general case) that the rowtype
of an inheritance child table is binary-compatible with the rowtype of
its parent, invent an expression node type that does the conversion
correctly.  Fixes the new bug exhibited by Kris Shannon as well as a
lot of old bugs that would only show up when using multiple inheritance
or after altering the parent table.

Allow planner to fold "stable" functions to constants when forming
selectivity estimates, per recent discussion.

In ALTER COLUMN TYPE, strip any implicit coercion operations appearing
at the top level of the column's old default expression before adding
an implicit coercion to the new column type.  This seems to satisfy the
principle of least surprise, as per discussion of bug #1290.

Fix problems with SQL functions returning rowtypes that have dropped
columns.  The returned tuple needs to have appropriate NULL columns
inserted so that it actually matches the declared rowtype.  It seemed
convenient to use a JunkFilter for this, so I made some cleanups and
simplifications in the JunkFilter code to allow it to support this
additional functionality.  (That in turn exposed a latent bug in
nodeAppend.c, which is that it was returning a tuple slot whose
descriptor didn't match its data.)  Also, move check_sql_fn_retval
out of pg_proc.c and into functions.c, where it seems to more naturally
belong.

Clean up handling of inherited-table update queries, per bug report
from Sebastian Böck.  The fix involves being more consistent about
when rangetable entries are copied or modified.  Someday we really
need to fix this stuff to not scribble on its input data structures
in the first place...

Pgindent run for 8.0.

Update copyright to 2004.

Standardize on the assumption that the arguments of a RowExpr correspond
to the physical layout of the rowtype, ie, there are dummy arguments
corresponding to any dropped columns in the rowtype.  We formerly had a
couple of places that did it this way and several others that did not.
Fixes Gaetano Mendola's "cache lookup failed for type 0" bug of 5-Aug.

Allow DECLARE CURSOR to take parameters from the portal in which it is
executed.  Previously, the DECLARE would succeed but subsequent FETCHes
would fail since the parameter values supplied to DECLARE were not
propagated to the portal created for the cursor.
In support of this, add type Oids to ParamListInfo entries, which seems
like a good idea anyway since code that extracts a value can double-check
that it got the type of value it was expecting.
Oliver Jowett, with minor editorialization by Tom Lane.

When using extended-query protocol, postpone planning of unnamed statements
until Bind is received, so that actual parameter values are visible to the
planner.  Make use of the parameter values for estimation purposes (but
don't fold them into the actual plan).  This buys back most of the
potential loss of plan quality that ensues from using out-of-line
parameters instead of putting literal values right into the query text.

This patch creates a notion of constant-folding expressions 'for
estimation purposes only', in which case we can be more aggressive than
the normal eval_const_expressions() logic can be.  Right now the only
difference in behavior is inserting bound values for Params, but it will
be interesting to look at other possibilities.  One that we've seen
come up repeatedly is reducing now() and related functions to current
values, so that queries like ... WHERE timestampcol > now() - '1 day'
have some chance of being planned effectively.

Oliver Jowett, with some kibitzing from Tom Lane.

Support assignment to subfields of composite columns in UPDATE and INSERT.
As a side effect, cause subscripts in INSERT targetlists to do something
more or less sensible; previously we evaluated such subscripts and then
effectively ignored them.  Another side effect is that UPDATE-ing an
element or slice of an array value that is NULL now produces a non-null
result, namely an array containing just the assigned-to positions.

Tweak palloc/repalloc to allow zero bytes to be requested, as per recent
proposal.  Eliminate several dozen now-unnecessary hacks to avoid palloc(0).
(It's likely there are more that I didn't find.)

Just about there on de-FastList-ification.

Use the new List API function names throughout the backend, and disable the
list compatibility API by default. While doing this, I decided to keep
the llast() macro around and introduce llast_int() and llast_oid() variants.

Reimplement the linked list data structure used throughout the backend.

In the past, we used a 'Lispy' linked list implementation: a "list" was
merely a pointer to the head node of the list. The problem with that
design is that it makes lappend() and length() linear time. This patch
fixes that problem (and others) by maintaining a count of the list
length and a pointer to the tail node along with each head node pointer.
A "list" is now a pointer to a structure containing some meta-data
about the list; the head and tail pointers in that structure refer
to ListCell structures that maintain the actual linked list of nodes.

The function names of the list API have also been changed to, I hope,
be more logically consistent. By default, the old function names are
still available; they will be disabled-by-default once the rest of
the tree has been updated to use the new API names.

Promote row expressions to full-fledged citizens of the expression syntax,
rather than allowing them only in a few special cases as before.  In
particular you can now pass a ROW() construct to a function that accepts
a rowtype parameter.  Internal generation of RowExprs fixes a number of
corner cases that used to not work very well, such as referencing the
whole-row result of a JOIN or subquery.  This represents a further step in
the work I started a month or so back to make rowtype values into
first-class citizens.

check_sql_fn_retval has always thought that we supported doing
'SELECT foo()' in a SQL function returning a rowtype, to simply pass
back the results of another function returning the same rowtype.
However, that hasn't actually worked in many years.  Now it works again.

Get rid of crocky use of RangeVar nodes in parser to represent partially
transformed whole-row variables.  Cleaner to use regular whole-row Vars.

Replace max_expr_depth parameter with a max_stack_depth parameter that
is measured in kilobytes and checked against actual physical execution
stack depth, as per my proposal of 30-Dec.  This gives us a fairly
bulletproof defense against crashing due to runaway recursive functions.

Revise syntax-error reporting behavior to give pleasant results for
errors in internally-generated queries, such as those submitted by
plpgsql functions.  Per recent discussions with Fabien Coelho.

Reimplement CASE val WHEN compval1 THEN ... WHEN compval2 THEN ... END
so that the 'val' is computed only once, per recent discussion.  The
speedup is not much when 'val' is just a simple variable, but could be
significant for larger expressions.  More importantly this avoids issues
with multiple evaluations of a volatile 'val', and it allows the CASE
expression to be reverse-listed in its original form by ruleutils.c.

Tweak planner so that index expressions and predicates are matched to
queries without regard to whether coercions are stated explicitly or
implicitly.  Per suggestion from Stephan Szabo.

simplify_function() mustn't try to evaluate functions that return
composite types, because TupleTableSlots aren't Datums and can't be
stored in Const nodes.  We can remove this restriction if we ever
adopt a cleaner runtime representation for whole-tuple results, but
at the moment it's broken.  Per example from Thomas Hallgren.

simplify_function() mustn't try to evaluate functions that return
composite types, because TupleTableSlots aren't Datums and can't be
stored in Const nodes.  We can remove this restriction if we ever
adopt a cleaner runtime representation for whole-tuple results, but
at the moment it's broken.  Per example from Thomas Hallgren.

Preserve AND/OR flatness during eval_const_expressions().  This seems a
useful improvement in any case, and it keeps the new logic for restrictinfo
structures happy.  Per report from Kris Jurka.

Improve has_nullable_targetlist() to allow strict functions of simple
variables, not just simple variables.  This was foreseen in the original
coding of this routine, but not implemented until now.  Responds to
performance gripe from Laurent Perez.

Improve UniquePath logic to detect the case where the input is already
known unique (eg, it is a SELECT DISTINCT ... subquery), and not do a
redundant unique-ification step.

Merge restrictlist_selectivity into clauselist_selectivity by
teaching the latter to accept either RestrictInfo nodes or bare
clause expressions; and cache the selectivity result in the RestrictInfo
node when possible.  This extends the caching behavior of approx_selectivity
to many more contexts, and should reduce duplicate selectivity
calculations.

Adjust the definition of RestrictInfo's left_relids and right_relids
fields: now they are valid whenever the clause is a binary opclause,
not only when it is a potential join clause (there is a new boolean
field canjoin to signal the latter condition).  This lets us avoid
recomputing the relid sets over and over while examining indexes.
Still more work to do to make this as useful as it could be, because
there are places that could use the info but don't have access to the
RestrictInfo node.

Clean up the usage of canonicalize_qual(): in particular, be consistent
about whether it is applied before or after eval_const_expressions().
I believe there were some corner cases where the system would fail to
recognize that a partial index is applicable because of the previous
inconsistency.  Store normal rather than 'implicit AND' representations
of constraints and index predicates in the catalogs.
initdb forced due to representation change of constraints/predicates.

query_tree_mutator should copy RangeTblEntry nodes even when it's not
planning to modify them itself.  Otherwise we end up with shared RTE
substructure, which breaks inheritance_planner because the rte->inh
flag needs to be independent in each copied subquery.  Per bug report
from Chris Piker.

query_tree_mutator should copy RangeTblEntry nodes even when it's not
planning to modify them itself.  Otherwise we end up with shared RTE
substructure, which breaks inheritance_planner because the rte->inh
flag needs to be independent in each copied subquery.  Per bug report
from Chris Piker.

$Header: -> $PostgreSQL Changes ...

Give a useful error message if a RangeVar is encountered in an expression.
Per example from Ian Barwick, 28-Oct-03.

Update this branch to match CVS head, includes WIN32 improvements.

Fix ARRAY[] construct so that in multidimensional case, elements can
be anything yielding an array of the proper kind, not only sub-ARRAY[]
constructs; do subscript checking at runtime not parse time.  Also,
adjust array_cat to make array || array comply with the SQL99 spec.

Joe Conway

Another pgindent run with updated typedefs.

Update copyrights to 2003.

pgindent run.

Tighten inline_function's test for overly complex parameters.  This
should catch most situations where repeated inlining blows up the
expression complexity unreasonably, as in Joe Conway's recent example.

Add error stack traceback support for SQL-language functions.

Error message editing in backend/optimizer, backend/rewrite.

Do honest transformation and preprocessing of LIMIT/OFFSET clauses,
instead of the former kluge whereby gram.y emitted already-transformed
expressions.  This is needed so that Params appearing in these clauses
actually work correctly.  I suppose some might claim that the side effect
of 'SELECT ... LIMIT 2+2' working is a new feature, but I say this is
a bug fix.

Code review for UPDATE tab SET col = DEFAULT patch ... whack it around
so it has some chance of working in rules ...

Dept. of second thoughts: supporting inlining of polymorphic SQL functions
takes only a few more lines of code than preventing it, so might as well
support it.

SQL functions can have arguments and results declared ANYARRAY or
ANYELEMENT.  The effect is to postpone typechecking of the function
body until runtime.  Documentation is still lacking.

Original patch by Joe Conway, modified to postpone type checking
by Tom Lane.

Support expressions of the form 'scalar op ANY (array)' and
'scalar op ALL (array)', where the operator is applied between the
lefthand scalar and each element of the array.  The operator must
yield boolean; the result of the construct is the OR or AND of the
per-element results, respectively.

Original coding by Joe Conway, after an idea of Peter's.  Rewritten
by Tom to keep the implementation strictly separate from subqueries.

Back out array mega-patch.

Joe Conway

Array mega-patch.

Joe Conway

Implement outer-level aggregates to conform to the SQL spec, with
extensions to support our historical behavior.  An aggregate belongs
to the closest query level of any of the variables in its argument,
or the current query level if there are no variables (e.g., COUNT(*)).
The implementation involves adding an agglevelsup field to Aggref,
and treating outer aggregates like outer variables at planning time.

Fix some planner performance problems with large WHERE clauses, by
introducing new 'FastList' list-construction subroutines to use in
hot spots.  This avoids the O(N^2) behavior of repeated lappend's
by keeping a tail pointer, while not changing behavior by reversing
list order as the lcons() method would do.

Replace functional-index facility with expressional indexes.  Any column
of an index can now be a computed expression instead of a simple variable.
Restrictions on expressions are the same as for predicates (only immutable
functions, no sub-selects).  This fixes problems recently introduced with
inlining SQL functions, because the inlining transformation is applied to
both expression trees so the planner can still match them up.  Along the
way, improve efficiency of handling index predicates (both predicates and
index expressions are now cached by the relcache) and fix 7.3 oversight
that didn't record dependencies of predicate expressions.

Infrastructure for deducing Param types from context, in the same way
that the types of untyped string-literal constants are deduced (ie,
when coerce_type is applied to 'em, that's what the type must be).
Remove the ancient hack of storing the input Param-types array as a
global variable, and put the info into ParseState instead.  This touches
a lot of files because of adjustment of routine parameter lists, but
it's really not a large patch.  Note: PREPARE statement still insists on
exact specification of parameter types, but that could easily be relaxed
now, if we wanted to do so.

Put back encoding-conversion step in processing of incoming queries;
I had inadvertently omitted it while rearranging things to support
length-counted incoming messages.  Also, change the parser's API back
to accepting a 'char *' query string instead of 'StringInfo', as the
latter wasn't buying us anything except overhead.  (I think when I put
it in I had some notion of making the parser API 8-bit-clean, but
seeing that flex depends on null-terminated input, that's not really
ever gonna happen.)

First phase of work on array improvements.  ARRAY[x,y,z] constructor
expressions, ARRAY(sub-SELECT) expressions, some array functions.
Polymorphic functions using ANYARRAY/ANYELEMENT argument and return
types.  Some regression tests in place, documentation is lacking.
Joe Conway, with some kibitzing from Tom Lane.

Adjust subquery qual pushdown rules to be more forgiving: if a qual
refers to a non-DISTINCT output column of a DISTINCT ON subquery, or
if it refers to a function-returning-set, we cannot push it down.
But the old implementation refused to push down *any* quals if the
subquery had any such 'dangerous' outputs.  Now we just look at the
output columns actually referenced by each qual expression.  More code
than before, but probably no slower since we don't make unnecessary checks.

Make eval_const_expressions simplify FieldSelect from a whole-row Var
into an ordinary one-field Var.  Per example from Chris Mungall.

Restructure parsetree representation of DECLARE CURSOR: now it's a
utility statement (DeclareCursorStmt) with a SELECT query dangling from
it, rather than a SELECT query with a few unusual fields in it.  Add
code to determine whether a planned query can safely be run backwards.
If DECLARE CURSOR specifies SCROLL, ensure that the plan can be run
backwards by adding a Materialize plan node if it can't.  Without SCROLL,
you get an error if you try to fetch backwards from a cursor that can't
handle it.  (There is still some discussion about what the exact
behavior should be, but this is necessary infrastructure in any case.)
Along the way, make EXPLAIN DECLARE CURSOR work.

COALESCE() and NULLIF() are now first-class expressions, not macros
that turn into CASE expressions.  They evaluate their arguments at most
once.  Patch by Kris Jurka, review and (very light) editorializing by me.

Create a distinction between Lists of integers and Lists of OIDs, to get
rid of the assumption that sizeof(Oid)==sizeof(int).  This is one small
step towards someday supporting 8-byte OIDs.  For the moment, it doesn't
do much except get rid of a lot of unsightly casts.

Replace planner's representation of relation sets, per pghackers discussion.
Instead of Lists of integers, we now store variable-length bitmap sets.
This should be faster as well as less error-prone.

Minor code cleanup: remove no-longer-useful pull_subplans() function,
and convert pull_agg_clause() into count_agg_clause(), which is a more
efficient way of doing what it's really being used for.

Determine the set of constraints applied to a domain at executor
startup, not in the parser; this allows ALTER DOMAIN to work correctly
with domain constraint operations stored in rules.  Rod Taylor;
code review by Tom Lane.

IN clauses appearing at top level of WHERE can now be handled as joins.
There are two implementation techniques: the executor understands a new
JOIN_IN jointype, which emits at most one matching row per left-hand row,
or the result of the IN's sub-select can be fed through a DISTINCT filter
and then joined as an ordinary relation.
Along the way, some minor code cleanup in the optimizer; notably, break
out most of the jointree-rearrangement preprocessing in planner.c and
put it in a new file prep/prepjointree.c.

Fix parse_agg.c to detect ungrouped Vars in sub-SELECTs; remove code
that used to do it in planner.  That was an ancient kluge that was
never satisfactory; errors should be detected at parse time when possible.
But at the time we didn't have the support mechanism (expression_tree_walker
et al) to make it convenient to do in the parser.

Adjust API of expression_tree_mutator and query_tree_mutator to
simplify callers.  It turns out the common case is that the caller
does want to recurse into sub-queries, so push support for that into
these subroutines.

Allow merge and hash joins to occur on arbitrary expressions (anything not
containing a volatile function), rather than only on 'Var = Var' clauses
as before.  This makes it practical to do flatten_join_alias_vars at the
start of planning, which in turn eliminates a bunch of klugery inside the
planner to deal with alias vars.  As a free side effect, we now detect
implied equality of non-Var expressions; for example in
	SELECT ... WHERE a.x = b.y and b.y = 42
we will deduce a.x = 42 and use that as a restriction qual on a.  Also,
we can remove the restriction introduced 12/5/02 to prevent pullup of
subqueries whose targetlists contain sublinks.
Still TODO: make statistical estimation routines in selfuncs.c and costsize.c
smarter about expressions that are more complex than plain Vars.  The need
for this is considerably greater now that we have to be able to estimate
the suitability of merge and hash join techniques on such expressions.

Further tweaking of parsetree & plantree representation of SubLinks.
Simplify SubLink by storing just a List of operator OIDs, instead of
a list of incomplete OpExprs --- that was a bizarre and bulky choice,
with no redeeming social value since we have to build new OpExprs
anyway when forming the plan tree.

Revise executor APIs so that all per-query state structure is built in
a per-query memory context created by CreateExecutorState --- and destroyed
by FreeExecutorState.  This provides a final solution to the longstanding
problem of memory leaked by various ExecEndNode calls.

Clean up plantree representation of SubPlan-s --- SubLink does not appear
in the planned representation of a subplan at all any more, only SubPlan.
This means subselect.c doesn't scribble on its input anymore, which seems
like a good thing; and there are no longer three different possible
interpretations of a SubLink.  Simplify node naming and improve comments
in primnodes.h.  No change to stored rules, though.

Phase 3 of read-only-plans project: ExecInitExpr now builds expression
execution state trees, and ExecEvalExpr takes an expression state tree
not an expression plan tree.  The plan tree is now read-only as far as
the executor is concerned.  Next step is to begin actually exploiting
this property.

Preliminary code review for domain CHECK constraints patch: add documentation,
make VALUE a non-reserved word again, use less invasive method of passing
ConstraintTestValue into transformExpr, fix problems with nested constraint
testing, do correct thing with NULL result from a constraint expression,
remove memory leak.  Domain checks still need much more work if we are going
to allow ALTER DOMAIN, however.

Phase 2 of read-only-plans project: restructure expression-tree nodes
so that all executable expression nodes inherit from a common supertype
Expr.  This is somewhat of an exercise in code purity rather than any
real functional advance, but getting rid of the extra Oper or Func node
formerly used in each operator or function call should provide at least
a little space and speed improvement.
initdb forced by changes in stored-rules representation.

Teach planner to expand sufficiently simple SQL-language functions
('SELECT expression') inline, like macros, during the constant-folding
phase of planning.  The actual expansion is not difficult, but checking
that we're not changing the semantics of the call turns out to be more
subtle than one might think; in particular must pay attention to
permissions issues, strictness, and volatility.

Code review for IS DISTINCT FROM patch.  Fix incorrect constant-folding
logic, dissuade planner from thinking that 'x IS DISTINCT FROM 42' may
be optimized into 'x = 42' (!!), cause dependency on = operator to be
recorded correctly, minor other improvements.

Use Params, rather than run-time-modified Const nodes, to handle
sublink results and COPY's domain constraint checking.  A Const that
isn't really constant is just a Bad Idea(tm).  Remove hacks in
parse_coerce and other places that were needed because of the former
klugery.

Remove unused constisset and constiscast fields of Const nodes.  Clean
up code and documentation associated with Param nodes.

Add DOMAIN check constraints.

Rod Taylor

Phase 2 of hashed-aggregation project.  nodeAgg.c now knows how to do
hashed aggregation, but there's not yet planner support for it.

Tweak querytree-dependency-extraction code so that columns of tables
that are explicitly JOINed are not considered dependencies unless they
are actually used in the query: mere presence in the joinaliasvars
list of a JOIN RTE doesn't count as being used.  The patch touches
a number of files because I needed to generalize the API of
query_tree_walker to support an additional flag bit, but the changes
are otherwise quite small.

pgindent run.

Code review for domain-constraints patch.  Use a new ConstraintTest node
type for runtime constraint checks, instead of misusing the parse-time
Constraint node for the purpose.  Fix some damage introduced into type
coercion logic; in particular ensure that a coerced expression tree will
read out the correct result type when inspected (patch had broken some
RelabelType cases).  Enforce domain NOT NULL constraints against columns
that are omitted from an INSERT.

oid is needed, it is added at the end of the struct (after the null
bitmap, if present).

Per Tom Lane's suggestion the information whether a tuple has an oid
or not is carried in the tuple descriptor.  For debugging reasons
tdhasoid is of type char, not bool.  There are predefined values for
WITHOID, WITHOUTOID and UNDEFOID.

This patch has been generated against a cvs snapshot from last week
and I don't expect it to apply cleanly to current sources.  While I
post it here for public review, I'm working on a new version against a
current snapshot.  (There's been heavy activity recently; hope to
catch up some day ...)

This is a long patch;  if it is too hard to swallow, I can provide it
in smaller pieces:

Part 1:  Accessor macros
Part 2:  tdhasoid in TupDesc
Part 3:  Regression test
Part 4:  Parameter withoid to heap_addheader
Part 5:  Eliminate t_oid from HeapTupleHeader

Part 2 is the most hairy part because of changes in the executor and
even in the parser;  the other parts are straightforward.

Up to part 4 the patched postmaster stays binary compatible to
databases created with an unpatched version.  Part 5 is small (100
lines) and finally breaks compatibility.

Manfred Koizar

Back out BETWEEN node patch, was causing initdb failure.

Finished the Between patch Christopher started.

Implements between (symmetric / asymmetric) as a node.

Executes the left or right expression once, makes a Const out of the
resulting Datum and executes the >=, <= portions out of the Const sets.

Of course, the parser does a fair amount of preparatory work for this to
happen.

Rod Taylor

I've fixed up the way domain constraints (not null and type length)
are managed as per request.

Moved from merging with table attributes to applying themselves during
coerce_type() and coerce_type_typmod.

Regression tests altered to test the cast() scenarios.

Rod Taylor

Implement the IS DISTINCT FROM operator per SQL99.
Reused the Expr node to hold DISTINCT which strongly resembles
 the existing OP info. Define DISTINCT_EXPR which strongly resembles
 the existing OPER_EXPR opType, but with handling for NULLs required
 by SQL99.
We have explicit support for single-element DISTINCT comparisons
 all the way through to the executor. But, multi-element DISTINCTs
 are handled by expanding into a comparison tree in gram.y as is done for
 other row comparisons. Per discussions, it might be desirable to move
 this into one or more purpose-built nodes to be handled in the backend.
Define the optional ROW keyword and token per SQL99.
 This allows single-element row constructs, which were formerly disallowed
 due to shift/reduce conflicts with parenthesized a_expr clauses.
Define the SQL99 TREAT() function. Currently, use as a synonym for CAST().

Update copyright to 2002.

Teach query_tree_walker, query_tree_mutator, and SS_finalize_plan to
process function RTE expressions, which they were previously missing.
This allows outer-Var references and subselects to work correctly in
the arguments of a function RTE.  Install check to prevent function RTEs
from cross-referencing Vars of sibling FROM-items, which doesn't make
any sense (if you want to join, write a JOIN or WHERE clause).

Get rid of long-since-vestigial Iter node type, in favor of adding a
returns-set boolean field in Func and Oper nodes.  This allows cleaner,
more reliable tests for expressions returning sets in the planner and
parser.  For example, a WHERE clause returning a set is now detected
and complained of in the parser, not only at runtime.

First pass at set-returning-functions in FROM, by Joe Conway with
some kibitzing from Tom Lane.  Not everything works yet, and there's
no documentation or regression test, but let's commit this so Joe
doesn't need to cope with tracking changes in so many files ...

Second try at fixing join alias variables.  Instead of attaching miscellaneous
lists to join RTEs, attach a list of Vars and COALESCE expressions that will
replace the join's alias variables during planning.  This simplifies
flatten_join_alias_vars while still making it easy to fix up varno references
when transforming the query tree.  Add regression test cases for interactions
of subqueries with outer joins.

Divide functions into three volatility classes (immutable, stable, and
volatile), rather than the old cachable/noncachable distinction.  This
allows indexscan optimizations in many places where we formerly didn't.
Also, add a pronamespace column to pg_proc (it doesn't do anything yet,
however).

First phase of SCHEMA changes, concentrating on fixing the grammar and
the parsetree representation.  As yet we don't *do* anything with schema
names, just drop 'em on the floor; but you can enter schema-compatible
command syntax, and there's even a primitive CREATE SCHEMA command.
No doc updates yet, except to note that you can now extract a field
from a function-returning-row's result with (foo(...)).fieldname.

Restructure representation of join alias variables.  An explicit JOIN
now has an RTE of its own, and references to its outputs now are Vars
referencing the JOIN RTE, rather than CASE-expressions.  This allows
reverse-listing in ruleutils.c to use the correct alias easily, rather
than painfully reverse-engineering the alias namespace as it used to do.
Also, nested FULL JOINs work correctly, because the result of the inner
joins are simple Vars that the planner can cope with.  This fixes a bug
reported a couple times now, notably by Tatsuo on 18-Nov-01.  The alias
Vars are expanded into COALESCE expressions where needed at the very end
of planning, rather than during parsing.
Also, beginnings of support for showing plan qualifier expressions in
EXPLAIN.  There are probably still cases that need work.
initdb forced due to change of stored-rule representation.

Shouldn't try to copy null datums with datumCopy.

Suppress subquery pullup and pushdown when the subquery has any
set-returning functions in its target list.  This ensures that we
won't rewrite the query in a way that places set-returning functions
into quals (WHERE clauses).  Cf. bug reports from Joe Conway.

New pgindent run with fixes suggested by Tom.  Patch manually reviewed,
initdb/regression tests pass.

Fix problems with subselects used in GROUP BY expressions, per gripe
from Philip Warner.  Side effect of change is that GROUP BY expressions
will not be re-evaluated at multiple plan levels anymore, whereas this
sometimes happened with old code.

pgindent run on all C files.  Java run to follow.  initdb/regression
tests pass.

Further thought shows that has_distinct_on_clause() needs to take much
more care with resjunk tlist entries than it was doing.  The original
coding ignored resjunk entries entirely, but a resjunk entry that is
in either the distinctClause or sortClause lists indicates that DISTINCT
ON was used.  It's important for ruleutils.c to get this right, else we
may dump views using DISTINCT ON incorrectly.

Fix optimizer to not try to push WHERE clauses down into a sub-SELECT that
has a DISTINCT ON clause, per bug report from Anthony Wood.  While at it,
improve the DISTINCT-ON-clause recognizer routine to not be fooled by out-
of-order DISTINCT lists.
Also, back-patch earlier fix to not push down into sub-SELECT with LIMIT.

Fix optimizer to not try to push WHERE clauses down into a sub-SELECT that
has a DISTINCT ON clause, per bug report from Anthony Wood.  While at it,
improve the DISTINCT-ON-clause recognizer routine to not be fooled by out-
of-order DISTINCT lists.

Add IS UNKNOWN, IS NOT UNKNOWN boolean tests, fix the existing boolean
tests to return the correct results per SQL9x when given NULL inputs.
Reimplement these tests as well as IS [NOT] NULL to have their own
expression node types, instead of depending on special functions.
From Joe Conway, with a little help from Tom Lane.

Modify optimizer data structures so that IndexOptInfo lists built for
create_index_paths are not immediately discarded, but are available for
subsequent planner work.  This allows avoiding redundant syscache lookups
in several places.  Change interface to operator selectivity estimation
procedures to allow faster and more flexible estimation.
Initdb forced due to change of pg_proc entries for selectivity functions!

Repair pgindent damage to comments.

pgindent run.  Make it all clean.

A subplan invoked within an aggregate function's argument should
be allowed to receive ungrouped variables of the current query level.
Curious that no one reported this bug before...

Cleanup

Add // -> /* */ mapping to pgindent.

Change Copyright from PostgreSQL, Inc to PostgreSQL Global Development Group.

Change SearchSysCache coding conventions so that a reference count is
maintained for each cache entry.  A cache entry will not be freed until
the matching ReleaseSysCache call has been executed.  This eliminates
worries about cache entries getting dropped while still in use.  See
my posting to pg-hackers of even date for more info.

Reimplementation of UNION/INTERSECT/EXCEPT.  INTERSECT/EXCEPT now meet the
SQL92 semantics, including support for ALL option.  All three can be used
in subqueries and views.  DISTINCT and ORDER BY work now in views, too.
This rewrite fixes many problems with cross-datatype UNIONs and INSERT/SELECT
where the SELECT yields different datatypes than the INSERT needs.  I did
that by making UNION subqueries and SELECT in INSERT be treated like
subselects-in-FROM, thereby allowing an extra level of targetlist where the
datatype conversions can be inserted safely.
INITDB NEEDED!

Subselects in FROM clause, per ISO syntax: FROM (SELECT ...) [AS] alias.
(Don't forget that an alias is required.)  Views reimplemented as expanding
to subselect-in-FROM.  Grouping, aggregates, DISTINCT in views actually
work now (he says optimistically).  No UNION support in subselects/views
yet, but I have some ideas about that.  Rule-related permissions checking
moved out of rewriter and into executor.
INITDB REQUIRED!

Use variable aliases, if supplied, rather than real column names in
complaints about ungrouped variables.  This is for consistency with
behavior elsewhere, notably the fact that the relname is reported as
an alias in these same complaints.  Also, it'll work with subselect-
in-FROM where old code didn't.

First cut at full support for OUTER JOINs.  There are still a few loose
ends to clean up (see my message of same date to pghackers), but mostly
it works.  INITDB REQUIRED!

SQL-language functions are now callable in ordinary fmgr contexts ...
for example, an SQL function can be used in a functional index.  (I make
no promises about speed, but it'll work ;-).)  Clean up and simplify
handling of functions returning sets.

fmgr interface mopup work.  Use new DatumGetBool and BoolGetDatum
macros where appropriate (the code used to have several different ways
of doing that, including Int32, Int8, UInt8, ...).  Remove last few
references to float32 and float64 typedefs --- it's all float4/float8
now.  The typedefs themselves should probably stay in c.h for a release
or two, though, to avoid breaking user-written C functions.

Clean up handling of variable-free qual clauses.  System now does the
right thing with variable-free clauses that contain noncachable functions,
such as 'WHERE random() < 0.5' --- these are evaluated once per
potential output tuple.  Expressions that contain only Params are
now candidates to be indexscan quals --- for example, 'var = ($1 + 1)'
can now be indexed.  Cope with RelabelType nodes atop potential indexscan
variables --- this oversight prevents 7.0.* from recognizing some
potentially indexscanable situations.

Remove 'func_tlist' from Func expression nodes, likewise 'param_tlist'
from Param nodes, per discussion a few days ago on pghackers.  Add new
expression node type FieldSelect that implements the functionality where
it's actually needed.  Clean up some other unused fields in Func nodes
as well.
NOTE: initdb forced due to change in stored expression trees for rules.

First stage of reclaiming memory in executor by resetting short-term
memory contexts.  Currently, only leaks in expressions executed as
quals or projections are handled.  Clean up some old dead cruft in
executor while at it --- unused fields in state nodes, that sort of thing.

Remove unused include files.  Do not touch /port or includes used by defines.

Constant-expression simplifier now knows how to simplify strict functions
that have at least one constant-NULL input, even if other inputs are
not constants.

get_relattval() should treat a NULL constant as a non-constant expression,
since it has no way to indicate to its caller that the constant is
actually NULL.  This prevents coredump in cases like
  WHERE textfield < null::text;

Ye-old pgindent run.  Same 4-space tabs.

Fix extremely nasty little bug observed when a sub-SELECT appears in
WHERE in a place where it can be part of a nestloop inner indexqual.
As the code stood, it put the same physical sub-Plan node into both
indxqual and indxqualorig of the IndexScan plan node.  That confused
later processing in the optimizer (which expected that tracing the
subPlan list would visit each subplan node exactly once), and would
probably have blown up in the executor if the planner hadn't choked first.
Fix by making the 'fixed' indexqual be a complete deep copy of the
original indexqual, rather than trying to share nodes below the topmost
operator node.  This had further ramifications though, because we were
making the aforesaid list of sub-Plan nodes during SS_process_sublinks
which is run before construction of the 'fixed' indexqual, meaning that
the copy of the sub-Plan didn't show up in that list.  Fix by rearranging
logic so that the sub-Plan list is built by the final set_plan_references
pass, not in SS_process_sublinks.  This may sound like a mess, but it's
actually a good deal cleaner now than it was before, because we are no
longer dependent on the assumption that planning will never make a copy
of a sub-Plan node.

Restructure planning code so that preprocessing of targetlist and quals
to simplify constant expressions and expand SubLink nodes into SubPlans
is done in a separate routine subquery_planner() that calls union_planner().
We formerly did most of this work in query_planner(), but that's the
wrong place because it may never see the real targetlist.  Splitting
union_planner into two routines also allows us to avoid redundant work
when union_planner is invoked recursively for UNION and inheritance
cases.  Upshot is that it is now possible to do something like
select float8(count(*)) / (select count(*) from int4_tbl)  from int4_tbl
group by f1;
which has never worked before.

Minor code rearrangement & doc improvement in eval_const_expressions().

Fix performance bug in constant-expression simplifier.  After finding
that the inputs to a given operator can be recursively simplified to
constants, it was evaluating the operator using the op's *original*
(unsimplified) arg list, so that any subexpressions had to be evaluated
again.  A constant subexpression at depth N got evaluated N times.
Probably not very important in practical situations, but it made us look
real slow in MySQL's 'crashme' test...

Create a new expression node type RelabelType, which exists solely to
represent the result of a binary-compatible type coercion.  At runtime
it just evaluates its argument --- but during type resolution, exprType
will pick up the output type of the RelabelType node instead of the type
of the argument.  This solves some longstanding problems with dropped
type coercions, an example being 'select now()::abstime::int4' which
used to produce date-formatted output, not an integer, because the
coercion to int4 was dropped on the floor.

Carry column aliases from the parser frontend. Enables queries like
  SELECT a FROM t1 tx (a);
Allow join syntax, including queries like
  SELECT * FROM t1 NATURAL JOIN t2;
Update RTE structure to hold column aliases in an Attr structure.

Add:

  * Portions Copyright (c) 1996-2000, PostgreSQL, Inc

to all files copyright Regents of Berkeley.  Man, that's a lot of files.

aggregate(DISTINCT ...) works, per SQL spec.
Note this forces initdb because of change of Aggref node in stored rules.

Replace generic 'Illegal use of aggregates' error message with one that
shows the specific ungrouped variable being complained of.  Perhaps this
will reduce user confusion...

Add system indexes to match all caches.
Make all system indexes unique.
Make all cache loads use system indexes.
Rename *rel to *relid in inheritance tables.
Rename cache names to be clearer.

Fix planner and rewriter to follow SQL semantics for tables that are
mentioned in FROM but not elsewhere in the query: such tables should be
joined over anyway.  Aside from being more standards-compliant, this allows
removal of some very ugly hacks for COUNT(*) processing.  Also, allow
HAVING clause without aggregate functions, since SQL does.  Clean up
CREATE RULE statement-list syntax the same way Bruce just fixed the
main stmtmulti production.
CAUTION: addition of a field to RangeTblEntry nodes breaks stored rules;
you will have to initdb if you have any rules.

Fix make_clause and make_opclause to record valid type info
in the Expr nodes they produce.  This fixes a few cases of errors like
'typeidTypeRelid: Invalid type - oid = 0' caused by calling parser-related
routines on expression trees that have already been processed by planner-
related routines.

Implement constant-expression simplification per Bernard
Frankpitt, plus some improvements from yours truly.  The simplifier depends
on the proiscachable field of pg_proc to tell it whether a function is
safe to pre-evaluate --- things like nextval() are not, for example.
Update pg_proc.h to contain reasonable cacheability information; as of
6.5.* hardly any functions were marked cacheable.  I may have erred too
far in the other direction; see recent mail to pghackers for more info.
This update does not force an initdb, exactly, but you won't see much
benefit from the simplifier until you do one.

Repair error noticed by Roberto Cornacchia: selectivity code
was rejecting negative attnums as bogus, which of course they are not.
Add code to get_attdisbursion to produce a useful value for OID attribute,
since VACUUM does not store stats for system attributes.
Also, repair bug that's been in eqjoinsel for a long time: it was taking
the max of the two columns' disbursions, whereas it should use the min.

Clean up some mistakes in handling of uplevel Vars in planner.
Most parts of the planner should ignore, or indeed never even see, uplevel
Vars because they will be or have been replaced by Params.  There were a
couple of places that got it wrong though, probably my fault from recent
changes...

Revise implementation of SubLinks so that there is a consistent,
documented intepretation of the lefthand and oper fields.  Fix a number of
obscure problems while at it --- for example, the old code failed if the parser
decided to insert a type-coercion function just below the operator of a
SubLink.
CAUTION: this will break stored rules that contain subplans.  You may
need to initdb.

Further planner/optimizer cleanups.  Move all set_tlist_references
and fix_opids processing to a single recursive pass over the plan tree
executed at the very tail end of planning, rather than haphazardly here
and there at different places.  Now that tlist Vars do not get modified
until the very end, it's possible to get rid of the klugy var_equal and
match_varid partial-matching routines, and just use plain equal()
throughout the optimizer.  This is a step towards allowing merge and
hash joins to be done on expressions instead of only Vars ...

Major planner/optimizer revision: get rid of PathOrder node type,
store all ordering information in pathkeys lists (which are now lists of
lists of PathKeyItem nodes, not just lists of lists of vars).  This was
a big win --- the code is smaller and IMHO more understandable than it
was, even though it handles more cases.  I believe the node changes will
not force an initdb for anyone; planner nodes don't show up in stored
rules.

Clean up optimizer's handling of indexscan quals that need to be
commuted (ie, the index var appears on the right).  These are now handled
the same way as merge and hash join quals that need to be commuted: the
actual reversing of the clause only happens if we actually choose the path
and generate a plan from it.  Furthermore, the clause is only reversed in
the 'indexqual' field of the plan, not in the 'indxqualorig' field.  This
allows the clause to still be recognized and removed from qpquals of upper
level join plans.  Also, simplify and generalize match_clause_to_indexkey;
now it recognizes binary-compatible indexes for join as well as restriction
clauses.

Minor cleanups and code beautification; eliminate some
routines that are now dead code.

Create a standardized expression_tree_mutator support routine
to go along with expression_tree_walker.  (_walker is not suitable for
routines that need to alter the tree structure significantly.)  Other minor
cleanups in clauses.c.

Final round before bed...more tomorrow...

First cut at doing LIKE/regex indexing optimization in
optimizer rather than parser.  This has many advantages, such as not
getting fooled by chance uses of operator names ~ and ~~ (the operators
are identified by OID now), and not creating useless comparison operations
in contexts where the comparisons will not actually be used as indexquals.
The new code also recognizes exact-match LIKE and regex patterns, and
produces an = indexqual instead of >= and <=.

This change does NOT fix the problem with non-ASCII locales: the code
still doesn't know how to generate an upper bound indexqual for non-ASCII
collation order.  But it's no worse than before, just the same deficiency
in a different place...

Also, dike out loc_restrictinfo fields in Plan nodes.  These were doing
nothing useful in the absence of 'expensive functions' optimization,
and they took a considerable amount of processing to fill in.

Further work on planning of indexscans.  Cleaned up interfaces
to index_selectivity so that it can be handed an indexqual clause list
rather than a bunch of assorted derivative data.

Clean up messy clause-selectivity code in clausesel.c; repair bug
identified by Hiroshi (incorrect cost attributed to OR clauses
after multiple passes through set_rest_selec()).  I think the code
was trying to allow selectivities of OR subclauses to be passed in
from outside, but noplace was actually passing any useful data, and
set_rest_selec() was passing wrong data.

Restructure representation of "indexqual" in IndexPath nodes so that
it is the same as for indxqual in completed IndexScan nodes: namely,
a toplevel list with an entry for each pass of the index scan, having
sublists that are implicitly-ANDed index qual conditions for that pass.
You don't want to know what the old representation was :-(

Improve documentation of OR-clause indexscan functions.

Remove useless 'notclause' field from RestrictInfo nodes.  (This might
force an initdb for anyone who has stored rules containing RestrictInfos,
but I do not think that RestrictInfo ever appears in completed plans.)

Final cleanup.

Change #include's to use <> and "" as appropriate.

Remove unused #includes in *.c files.

On second thought, expression_tree_walker should handle bare
SubLink nodes after all ...

Create a generic expression-tree-walker subroutine, which
will gradually replace all of the boilerplate tree-walk-recursion code that
currently exists in O(N) slightly different forms in N subroutines.
I've had it with adding missing cases to these subroutines...

Another pgindent run.  Sorry folks.

pgindent run over code.

Change error messages to oids come out as %u and not %d.  Change has no
real affect now.

Executor no longer cares about mergejoinop, mergerightorder, mergeleftorder,
so remove them from MergeJoin node.  Hack together a partial
solution for commuted mergejoin operators --- yesterday
a mergejoin int4 = int8 would crash if the planner decided to
commute it, today it works.  The planner's representation of
mergejoins really needs a rewrite though.
Also, further testing of mergejoin ops in opr_sanity regress test.

Fix bushy plans.  Cleanup.

Fix optimizer coredump with unary-operator WHERE clauses.
A test case is:
create table linetab (x line);
select * from linetab where ?| x;
which coredumps in 6.4.2 and current sources.

Change my-function-name-- to my_function_name, and optimizer renames.

Cleanup of source files where 'return' or 'var =' is alone on a line.

Rename Aggreg to Aggref.

Implement CASE expression.

New HeapTuple structure/interface.

OK, folks, here is the pgindent output.

Renaming cleanup, no pgindent yet.

Fix for regproc

OR clause index fix

Cleanup optimizer function names and clarify code.

New pgindent.

OPTIMIZER_DEBUG additions.

Remove un-needed braces around single statements.

pgindent run before 6.3 release, with Thomas' requested changes.

Support for subselects.

Add Var.varlevelup to code.  More parser cleanup.

Fix for select 1=1 or 2=2, select 1=1 and 2=2, and select sum(2+2).

Fix handling of functions with non-attribute first argument in WHERE
clauses.

Used modified version of indent that understands over 100 typedefs.

Add typdefs to pgindent run.

Another PGINDENT run that changes variable indenting and case label indenting.  Also static variable indenting.

Massive commit to run PGINDENT on all *.c and *.h files.

Make functions static where possible, enclose unused functions in #ifdef NOT_USED.

INDEXSCAN_PATCH changes: should work for (Param OP VAR) too
(but I didn't test FUNC OP PARAM and PARAM OP FUNC - no time)

As someone asked for this feature - patch for 1.09 follows.
Now You can do queries like

select sum(some_func(x)) from ...
select min(table1.x + table2.y) from table1, table2 where ...

and so on.

Vadim

More files cleaned out

Fixes:

The problem is that the function arguments are not considered as possible key
candidates for index scan and so only a sequential scan is possible inside
the body of a function.  I have therefore made some patches to the optimizer
so that indices are now used also by functions.  I have also moved the plan
debug message from pg_eval to pg_plan so that it is printed also for plans
genereated for function execution.  I had also to add an index rescan to the
executor because it ignored the parameters set in the execution state, they
were flagged as runtime variables in ExecInitIndexScan but then never used
by the executor so that the scan were always done with any key=1. Very odd.
This means that an index rescan is now done twice for each function execution
which uses an index, the first time when the index scan is initialized and
the second when the actual function arguments are finally available for the
execution.  I don't know what is the cost of an double index scan but I
suppose it is anyway less than the cost of a full sequential scan, at leat
for large tables. This is my patch, you must also add -DINDEXSCAN_PATCH in
Makefile.global to enable the changes.

Submitted by: Massimo Dal Zotto <dz@cs.unitn.it>

Fix applied for NESTLOOP bug

submitted by: vadim@sable.krasnoyarsk.su (Vadim B. Mikheev)

Postgres95 1.01 Distribution - Virgin Sources

Initial revision