<para><link linkend="multimaster-usage">Limitations</link></para>

alive and reachable by other nodes. To be configured with <filename>multimaster</filename>, the cluster must include at least two nodes. In most cases, three

cluster nodes are enough to ensure high availability. Since the data on all cluster nodes is the same, you do not typically need more than five cluster nodes.</para>

<para>When a failed node

Write-Ahead Log (<acronym>WAL</acronym>) data in the corresponding replication slot. If <acronym>WAL</acronym> data is no longer available for the time when the node was excluded from the cluster, you can <link linkend="multimaster-restoring-a-node-manually">restore the node using <application>pg_basebackup</application></link>.

<sect2 id="multimaster-usage">

<title>Limitations</title>

<para>The <filename>multimaster</filename> extension takes care of the database replication in a fully automated way. You can perform write transactions on any node and work with temporary tables on each cluster node simultaneosly. However, make sure to take the following replication restrictions into account:</para>

<itemizedlist>

<listitem>

<para>

<filename>multimaster</filename> can only replicate one database

per cluster, which is specified in the <varname>multimaster.conn_strings</varname> variable. If you try to connect to a different database, <filename>multimaster</filename> will return a corresponding error message.

</para>

</listitem>

<listitem>

<para>

The replicated tables must have primary keys or replica identity. Otherwise,

<filename>multimaster</filename> will not allow replication

because of the logical replication restrictions. Unlogged tables are not replicated, as in the standard <productname>PostgreSQL</productname>.

</para>

<note><para>You can enable replication

of tables without primary keys by setting the <varname>multimaster.ignore_tables_without_pk</varname> variable to <literal>false</literal>. However, take into account that

<filename>multimaster</filename> does not allow update operations on such tables.</para></note>

</listitem>

<listitem>

<para>

Isolation level. The <filename>multimaster</filename> extension

supports <emphasis><literal>read committed</literal></emphasis> and <emphasis><literal>repeatable read</literal></emphasis> isolation levels. <emphasis><literal>Serializable</literal></emphasis> isolation level is currently not supported.</para>

<important>

<para>Using <literal>repeatable read</literal> isolation level increases

the probability of serialization failure at commit time. Unlike in the standard <productname>PostgreSQL</productname>, <literal>read committed</literal> level can also cause serialization failures on a multi-master cluster.</para>

<para>When performing a write transaction, <filename>multimaster</filename> blocks the affected objects only on the node on which the transaction is performed. However, since write transactions are allowed on all nodes, other transactions can try to change the same objects on the neighbor nodes at the same time. In this case, the replication of the first transaction can fail because the affected objects on the neighbor nodes are already blocked by another transaction. Similarly, the latter transaction cannot be replicated to the first node. In this case, a distributed deadlock occurs. As a result, one of the transactions is automatically rolled back and needs to be repeated. The application must be ready to retry transactions.

</para>

<para>If your typical workload has too many rollbacks, it is recommended to use <literal>read committed</literal> isolation level. However, the <literal>read committed</literal> still does not guarantee the absence of deadlocks on a multi-master cluster. If using the <literal>read committed</literal> level does not help, you can try directing all the write transactions to a single node.</para>

</important>

</listitem>

<listitem>

<para>

Sequence generation. To avoid conflicts between unique identifiers on different nodes, <filename>multimaster</filename> modifies the default behavior of sequence generators. For each node, ID generation is started with the node number and is incremented by the number of nodes. For example, in a three-node cluster, 1, 4, and 7 IDs are allocated to the objects written onto the first node, while 2, 5, and 8 IDs are reserved for the second node. If you change the number of nodes in the cluster, the incrementation interval for new IDs is adjusted accordingly. Thus, the generated sequence values are not monotonic.

</para>

</listitem>

<listitem>

<para>

<acronym>DDL</acronym> replication. While <filename>multimaster</filename>

replicates data on the logical level, <acronym>DDL</acronym> is replicated on the

statement level, which results in distributed commits of the same

statement on different nodes. As a result, complex <acronym>DDL</acronym>

scenarios, such as stored procedures and temporary tables, may

work differently as compared to the standard <productname>PostgreSQL</productname>.

</para>

</listitem>

<listitem>

<para>

Commit latency. In the current implementation of logical

replication, <filename>multimaster</filename> sends data to subscriber nodes only after the

local commit, so you have to wait for transaction processing twice: first on the local node,

and then on all the other nodes simultaneously. In the case of a heavy-write transaction, this may result in a noticeable delay.

</para>

</listitem>

</itemizedlist>

<para>If you have any data that must be present on one of the nodes only, you can exclude a particular table from replication, as follows:

<programlisting>SELECT * FROM <function>mtm.make_table_local</function>(::regclass::oid) </programlisting>

where <literal>regclass</literal> is the name of the table and <literal>oid</literal> is the unique table identifier.</para>

</sect2>

When <productname>&productname;</productname> loads the <filename>multimaster</filename> shared

<para><filename>multimaster</filename> currently supports the <literal>read committed</literal> and <literal>repeatable read</literal> isolation levels only, which can cause unexpected serialization failures in your workload. For details, see <xref linkend="multimaster-usage">.</para>

transaction. The <literal>PRECOMMITTED</literal> messages help

avoid the situation when the crashed node has already committed

or aborted the transaction, but has not notified other nodes

<para><link linkend="multimaster-restoring-a-node-manually">Restoring a Cluster Node</link></para>

extensions.

configure the cluster with <filename>multimaster</filename>.</para>

<para>Suppose

<para>The <literal>max_nodes</literal> variable defines the cluster size. In most cases, three cluster nodes are enough to ensure high availability. Since the data on all cluster nodes is the same, you do not typically need more than five cluster nodes.</para>

The maximum number of nodes allowed in the cluster. In most cases, three cluster nodes are enough to ensure high availability. Since the data on all cluster nodes is the same, you do not typically need more than five cluster nodes. The maximum possible number of nodes is limited to 64.</para><para>Default: the number of nodes specified in the <varname>multimaster.conn_strings</varname> variable