diff --git a/README.md b/README.md index e1d35ccc..7f0483bb 100644 --- a/README.md +++ b/README.md @@ -59,7 +59,7 @@ Examples can be found in the [/samples][samples] directory and the ## Help -Questions can be asked in [Github Discussions][ghdiscussions]. +Questions can be asked in [GitHub Discussions][ghdiscussions]. Problem reports can be raised in [GitHub Issues][ghissues]. diff --git a/doc/src/api_manual/module.rst b/doc/src/api_manual/module.rst index 59c8d2b5..45a33682 100644 --- a/doc/src/api_manual/module.rst +++ b/doc/src/api_manual/module.rst @@ -2664,7 +2664,7 @@ Oracledb Methods are parsed by python-oracledb itself and a generated connect descriptor is sent to the Oracle Client libraries. This value is only used in the python-oracledb Thick mode. The default value is - :attr:`defualts.thick_mode_dsn_passthrough`. For more information, see + :attr:`defaults.thick_mode_dsn_passthrough`. For more information, see :ref:`usingconfigfiles`. The ``extra_auth_params`` parameter is expected to be a dictionary diff --git a/doc/src/release_notes.rst b/doc/src/release_notes.rst index ef0acd26..c32207bf 100644 --- a/doc/src/release_notes.rst +++ b/doc/src/release_notes.rst @@ -11,6 +11,42 @@ Release changes are listed as affecting Thin Mode (the default runtime behavior of python-oracledb), as affecting the optional :ref:`Thick Mode `, or as being 'Common' for changes that impact both modes. +oracledb 3.1.1 (May 2025) +------------------------- + +Thin Mode Changes ++++++++++++++++++ + +#) Fixed bug with :meth:`Connection.is_healthy()` after a session is killed, + such as by a DBA running ALTER SYSTEM KILL SESSION. Previously, in some + databases, it could incorrectly return *True*, while in other cases it + could hang. + +Common Changes +++++++++++++++ + +#) Added support for using the Cython 3.1 release + (`issue 493 `__). +#) Improvements to data frame fetching with :meth:`Connection.fetch_df_all()` + and :meth:`Connection.fetch_df_batches()`: + + - Added support for converting an :ref:`OracleDataFrame + ` object to a foreign data frame object more than + once + (`issue 470 `__). + - Fixed a bug resulting in a segfault when attempting to use an + :ref:`output type handler ` while fetching data frames + (`issue 486 `__). + - Fixed memory corruption in data frame queries + (`issue 489 `__). + +#) Fixed parsing of the connection string in the + :ref:`Azure App Centralized Configuration Provider + `. +#) Miscellaneous grammar and spelling fixes by John Bampton + (`PR 479 `__). + + oracledb 3.1.0 (April 2025) --------------------------- @@ -323,7 +359,7 @@ Thin Mode Changes connection string. #) Added :meth:`oracledb.enable_thin_mode()` as a means of enabling python-oracledb Thin mode without waiting for an initial connection to be - succesfully established. Since python-oracledb defaults to Thin mode, this + successfully established. Since python-oracledb defaults to Thin mode, this method is mostly useful for applications with multiple threads concurrently creating connections to databases when the application starts (`issue 408 `__). @@ -1700,7 +1736,7 @@ cx_Oracle 8.2 (May 2021) connection. #) Eliminated a memory leak when calling :meth:`SodaOperation.filter()` with a dictionary. -#) The distributed transaction handle assosciated with the connection is now +#) The distributed transaction handle associated with the connection is now cleared on commit or rollback (`issue 530 `__). #) Added a check to ensure that when setting variables or object attributes, diff --git a/doc/src/user_guide/appendix_b.rst b/doc/src/user_guide/appendix_b.rst index 98748205..f2725fea 100644 --- a/doc/src/user_guide/appendix_b.rst +++ b/doc/src/user_guide/appendix_b.rst @@ -148,7 +148,7 @@ differs from the python-oracledb Thick mode in the following ways: ``handle`` parameters. The parameters that are ignored in the Thick mode include ``wallet_password``, ``disable_oob``, and ``debug_jdwp`` parameters. -* The python-oracledb Thin mode only suppports :ref:`homogeneous +* The python-oracledb Thin mode only supports :ref:`homogeneous ` pools. * The python-oracledb Thin mode creates connections in a daemon thread and so diff --git a/doc/src/user_guide/connection_handling.rst b/doc/src/user_guide/connection_handling.rst index 95d0d788..c57701cd 100644 --- a/doc/src/user_guide/connection_handling.rst +++ b/doc/src/user_guide/connection_handling.rst @@ -2448,7 +2448,7 @@ The :meth:`Connection.is_healthy()` method is an alternative to it does not perform a full connection check. If the ``getmode`` parameter in :meth:`oracledb.create_pool()` is set to -:data:`oracledb.POOL_GETMODE_TIMEDWAIT`, then the maxium amount of time an +:data:`oracledb.POOL_GETMODE_TIMEDWAIT`, then the maximum amount of time an :meth:`~ConnectionPool.acquire()` call will wait to get a connection from the pool is limited by the value of the :data:`ConnectionPool.wait_timeout` parameter. A call that cannot be immediately satisfied will wait no longer diff --git a/doc/src/user_guide/exception_handling.rst b/doc/src/user_guide/exception_handling.rst index 6fe6ad6b..1f715a20 100644 --- a/doc/src/user_guide/exception_handling.rst +++ b/doc/src/user_guide/exception_handling.rst @@ -84,7 +84,7 @@ in the examples below: DPY-4010: a bind variable replacement value for placeholder ":1" was not provided * Connection messages: The python-oracledb Thin mode connection and networking - is handled by Python itself. Some errors portable accross operating systems + is handled by Python itself. Some errors portable across operating systems and Python versions have DPY-prefixed errors displayed by python-oracledb. Other messages are returned directly from Python and may vary accordingly. The traditional Oracle connection errors with prefix "ORA" are not shown. For diff --git a/doc/src/user_guide/extending.rst b/doc/src/user_guide/extending.rst index 313c8a1f..0d0b31b2 100644 --- a/doc/src/user_guide/extending.rst +++ b/doc/src/user_guide/extending.rst @@ -16,7 +16,7 @@ Subclassing Connections ======================= Subclassing enables applications to change python-oracledb, for example by -extending connection and statement execution behvior. This can be used to +extending connection and statement execution behavior. This can be used to alter, or log, connection and execution parameters, or to further change python-oracledb functionality. @@ -220,7 +220,7 @@ strings prefixed with "myprefix://". In myhookfunc: protocol=myprefix arg=localhost/orclpdb1 host=localhost, port=1521, service name=orclpdb1 -7. To uninstall the plugin, simply remove the packge:: +7. To uninstall the plugin, simply remove the package:: python -m pip uninstall myplugin diff --git a/doc/src/user_guide/initialization.rst b/doc/src/user_guide/initialization.rst index f057f8a9..bb30a61b 100644 --- a/doc/src/user_guide/initialization.rst +++ b/doc/src/user_guide/initialization.rst @@ -317,7 +317,7 @@ going to be used. In one special case, you may wish to explicitly enable Thin mode to prevent Thick mode from being enabled later. To allow application portability, the driver's internal logic allows -applications to initally attempt :ref:`standalone connection +applications to initially attempt :ref:`standalone connection ` creation in Thin mode, but then lets them :ref:`enable Thick mode ` if that connection is unsuccessful. An example is when trying to connect to an Oracle Database that turns out to be an old diff --git a/samples/bind_insert.py b/samples/bind_insert.py index abd7750f..712e858e 100644 --- a/samples/bind_insert.py +++ b/samples/bind_insert.py @@ -86,7 +86,7 @@ # Inserting a single bind still needs tuples # ----------------------------------------------------------------------------- -rows = [("Eleventh",), ("Twelth",)] +rows = [("Eleventh",), ("Twelfth",)] with connection.cursor() as cursor: cursor.executemany("insert into mytab(id, data) values (12, :1)", rows) diff --git a/samples/bind_insert_async.py b/samples/bind_insert_async.py index 2e3a3660..37f0acfb 100644 --- a/samples/bind_insert_async.py +++ b/samples/bind_insert_async.py @@ -92,7 +92,7 @@ async def main(): # Inserting a single bind still needs tuples # ------------------------------------------------------------------------- - rows = [("Eleventh",), ("Twelth",)] + rows = [("Eleventh",), ("Twelfth",)] await connection.executemany( "insert into mytab(id, data) values (12, :1)", rows diff --git a/samples/containers/app_dev/README.md b/samples/containers/app_dev/README.md index f05d3257..25569ebd 100644 --- a/samples/containers/app_dev/README.md +++ b/samples/containers/app_dev/README.md @@ -23,7 +23,7 @@ It has been tested on macOS using podman and docker. By default, Apache has SSL enabled and is listening on port 8443. -## Usage for Application Devlopment +## Usage for Application Development - Run a container: diff --git a/samples/cqn.py b/samples/cqn.py index c5c6516a..d6399f75 100644 --- a/samples/cqn.py +++ b/samples/cqn.py @@ -55,7 +55,7 @@ def callback(message): registered = False return print("Message database name:", message.dbname) - print("Message tranasction id:", message.txid) + print("Message transaction id:", message.txid) print("Message queries:") for query in message.queries: print("--> Query ID:", query.id) diff --git a/samples/database_change_notification.py b/samples/database_change_notification.py index 7861ab2c..385e73bf 100644 --- a/samples/database_change_notification.py +++ b/samples/database_change_notification.py @@ -55,7 +55,7 @@ def callback(message): registered = False return print("Message database name:", message.dbname) - print("Message tranasction id:", message.txid) + print("Message transaction id:", message.txid) print("Message tables:") for table in message.tables: print("--> Table Name:", table.name) diff --git a/samples/json_blob.py b/samples/json_blob.py index 61b8576f..e0d06658 100644 --- a/samples/json_blob.py +++ b/samples/json_blob.py @@ -57,7 +57,7 @@ client_version = oracledb.clientversion()[0] db_version = int(connection.version.split(".")[0]) -# Minimum database vesion is 12 +# Minimum database version is 12 if db_version < 12: sys.exit("This example requires Oracle Database 12.1.0.2 or later") diff --git a/samples/json_blob_async.py b/samples/json_blob_async.py index d8b9221d..5c9fb45c 100644 --- a/samples/json_blob_async.py +++ b/samples/json_blob_async.py @@ -54,7 +54,7 @@ async def main(): params=sample_env.get_connect_params(), ) - # Minimum database vesion is 12 + # Minimum database version is 12 db_version = int(connection.version.split(".")[0]) if db_version < 12: sys.exit("This example requires Oracle Database 12.1.0.2 or later") diff --git a/samples/tutorial/setup_tutorial.py b/samples/tutorial/setup_tutorial.py index 3a64cc7c..64ba3c2a 100644 --- a/samples/tutorial/setup_tutorial.py +++ b/samples/tutorial/setup_tutorial.py @@ -35,7 +35,7 @@ user=db_config.user, password=db_config.pw, dsn=db_config.dsn ) -# create sample schemas and defintions for the tutorial +# create sample schemas and definitions for the tutorial print("Setting up the sample tables and other DB objects for the tutorial...") run_sql_script.run_sql_script( con, "setup_tutorial", user=db_config.user, pw=db_config.pw diff --git a/setup.py b/setup.py index 9729f381..9e753737 100644 --- a/setup.py +++ b/setup.py @@ -78,7 +78,7 @@ thin_depends.append(base_pxd) # if the platform is macOS: -# - target the minimim OS version that current Python packages work with. +# - target the minimum OS version that current Python packages work with. # (Use 'otool -l /path/to/python' and look for 'version' in the # LC_VERSION_MIN_MACOSX section) # - add argument required for cross-compilation for both x86_64 and arm64 diff --git a/src/oracledb/driver_mode.py b/src/oracledb/driver_mode.py index 630b9934..42594a9f 100644 --- a/src/oracledb/driver_mode.py +++ b/src/oracledb/driver_mode.py @@ -127,7 +127,7 @@ def is_thin_mode() -> bool: oracledb.init_oracle_client() is called successfully, then a subsequent call to is_thin_mode() will return False indicating that Thick mode is enabled. Once the first standalone connection or connection pool is - created succesfully, or a call to oracledb.init_oracle_client() is made + created successfully, or a call to oracledb.init_oracle_client() is made successfully, then python-oracledb's mode is fixed and the value returned by is_thin_mode() will never change for the lifetime of the process. diff --git a/src/oracledb/errors.py b/src/oracledb/errors.py index c3eec753..9ef74320 100644 --- a/src/oracledb/errors.py +++ b/src/oracledb/errors.py @@ -112,17 +112,21 @@ def _make_adjustments(self): args = {} if match is None else match.groupdict() else: driver_error_num = driver_error_info - if driver_error_num == ERR_CONNECTION_CLOSED: - self.is_session_dead = True driver_error = _get_error_text(driver_error_num, **args) self.message = f"{driver_error}\n{self.message}" self.full_code = f"{ERR_PREFIX}-{driver_error_num:04}" # determine exception class to use when raising this error + # also determine whether error is recoverable and whether the session + # is deemed "dead" if self.full_code.startswith("DPY-"): driver_error_num = int(self.full_code[4:]) + if driver_error_num == ERR_CONNECTION_CLOSED: + self.is_session_dead = self.isrecoverable = True self.exc_type = ERR_EXCEPTION_TYPES[driver_error_num // 1000] elif self.code != 0: + if self.code in ERR_RECOVERABLE_ERROR_CODES: + self.isrecoverable = True if self.code in ERR_INTEGRITY_ERROR_CODES: self.exc_type = exceptions.IntegrityError elif self.code in ERR_INTERFACE_ERROR_CODES: @@ -485,6 +489,21 @@ def _raise_not_supported(feature: str) -> None: 28511, # lost RPC connection to heterogeneous remote agent ] +# Oracle error codes that are deemed recoverable +# NOTE: this does not include the errors that are mapped to +# ERR_CONNECTION_CLOSED since those are all deemed recoverable +ERR_RECOVERABLE_ERROR_CODES = [ + 376, # file %s cannot be read at this time + 1033, # ORACLE initialization or shutdown in progress + 1034, # the Oracle instance is not available for use + 1090, # shutdown in progress + 1115, # IO error reading block from file %s (block # %s) + 12514, # Service %s is not registered with the listener + 12571, # TNS:packet writer failure + 12757, # instance does not currently know of requested service + 16456, # missing or invalid value +] + # driver error message exception types (multiples of 1000) ERR_EXCEPTION_TYPES = { 1: exceptions.InterfaceError, diff --git a/src/oracledb/impl/base/cursor.pyx b/src/oracledb/impl/base/cursor.pyx index 3a78fa40..5d6b11d5 100644 --- a/src/oracledb/impl/base/cursor.pyx +++ b/src/oracledb/impl/base/cursor.pyx @@ -301,11 +301,15 @@ cdef class BaseCursorImpl: """ Return the output type handler to use for the cursor. If one is not directly defined on the cursor then the one defined on the connection - is used instead. + is used instead. When fetching Arrow data, however, no output type + handlers are used since for most data no conversion to Python objects + ever takes place. """ cdef: BaseConnImpl conn_impl object type_handler + if self.fetching_arrow: + return None if self.outputtypehandler is not None: type_handler = self.outputtypehandler else: diff --git a/src/oracledb/impl/thin/messages/base.pyx b/src/oracledb/impl/thin/messages/base.pyx index 9e1efb9c..83c05075 100644 --- a/src/oracledb/impl/thin/messages/base.pyx +++ b/src/oracledb/impl/thin/messages/base.pyx @@ -67,39 +67,10 @@ cdef class Message: connection" error is detected, the connection is forced closed immediately. """ - cdef bint is_recoverable = False if self.error_occurred: - if self.error_info.num in ( - 28, # session has been terminated - 31, # session marked for kill - 376, # file %s cannot be read at this time - 603, # ORACLE server session terminated - 1012, # not logged on - 1033, # ORACLE initialization or shutdown in progress - 1034, # the Oracle instance is not available for use - 1089, # immediate shutdown or close in progress - 1090, # shutdown in progress - 1092, # ORACLE instance terminated - 1115, # IO error reading block from file %s (block # %s) - 2396, # exceeded maximum idle time - 3113, # end-of-file on communication channel - 3114, # not connected to ORACLE - 3135, # connection lost contact - 12153, # TNS:not connected - 12514, # Service %s is not registered with the listener - 12537, # TNS:connection closed - 12547, # TNS:lost contact - 12570, # TNS:packet reader failure - 12571, # TNS:packet writer failure - 12583, # TNS:no reader - 12757, # instance does not currently know of requested service - 16456, # missing or invalid value - ): - is_recoverable = True error = errors._Error(self.error_info.message, code=self.error_info.num, - offset=self.error_info.pos, - isrecoverable=is_recoverable) + offset=self.error_info.pos) if error.is_session_dead: self.conn_impl._protocol._force_close() raise error.exc_type(error) diff --git a/src/oracledb/impl/thin/packet.pyx b/src/oracledb/impl/thin/packet.pyx index 21de9b62..4a2e907d 100644 --- a/src/oracledb/impl/thin/packet.pyx +++ b/src/oracledb/impl/thin/packet.pyx @@ -65,7 +65,7 @@ cdef class Packet: char *ptr ptr = cpython.PyBytes_AS_STRING(self.buf) flags = decode_uint16be( &ptr[PACKET_HEADER_SIZE]) - if flags & TNS_DATA_FLAGS_END_OF_RESPONSE: + if flags & TNS_DATA_FLAGS_END_OF_RESPONSE or flags & TNS_DATA_FLAGS_EOF: return True if self.packet_size == PACKET_HEADER_SIZE + 3 \ and ptr[PACKET_HEADER_SIZE + 2] == TNS_MSG_TYPE_END_OF_RESPONSE: @@ -231,6 +231,8 @@ cdef class ReadBuffer(Buffer): errors._raise_err(errors.ERR_UNSUPPORTED_INBAND_NOTIFICATION, err_num=self._pending_error_num) elif self._transport is None: + if self._pending_error_num == TNS_ERR_SESSION_SHUTDOWN: + errors._raise_err(errors.ERR_CONNECTION_CLOSED) errors._raise_err(errors.ERR_NOT_CONNECTED) cdef int _get_int_length_and_sign(self, uint8_t *length, diff --git a/src/oracledb/impl/thin/protocol.pyx b/src/oracledb/impl/thin/protocol.pyx index 63692de4..1dfbbb11 100644 --- a/src/oracledb/impl/thin/protocol.pyx +++ b/src/oracledb/impl/thin/protocol.pyx @@ -902,6 +902,9 @@ cdef class BaseAsyncProtocol(BaseProtocol): """ if not self._in_connect: self._transport = None + self._read_buf._transport = None + self._write_buf._transport = None + self._read_buf._pending_error_num = TNS_ERR_SESSION_SHUTDOWN if self._read_buf._waiter is not None \ and not self._read_buf._waiter.done(): error = errors._create_err(errors.ERR_CONNECTION_CLOSED) diff --git a/src/oracledb/interchange/column.py b/src/oracledb/interchange/column.py index 8701b7b4..c44873dc 100644 --- a/src/oracledb/interchange/column.py +++ b/src/oracledb/interchange/column.py @@ -41,19 +41,8 @@ ) from .nanoarrow_bridge import ( - NANOARROW_TIME_UNIT_SECOND, - NANOARROW_TIME_UNIT_MILLI, - NANOARROW_TIME_UNIT_MICRO, - NANOARROW_TIME_UNIT_NANO, - NANOARROW_TYPE_BINARY, - NANOARROW_TYPE_DOUBLE, - NANOARROW_TYPE_FLOAT, - NANOARROW_TYPE_INT64, - NANOARROW_TYPE_LARGE_BINARY, - NANOARROW_TYPE_LARGE_STRING, - NANOARROW_TYPE_STRING, - NANOARROW_TYPE_TIMESTAMP, - NANOARROW_TYPE_DECIMAL128, + ArrowTimeUnit, + ArrowType, ) @@ -92,8 +81,8 @@ def _offsets_buffer(self): size_in_bytes=size_bytes, address=address, buffer_type="offsets" ) if self.ora_arrow_array.arrow_type in ( - NANOARROW_TYPE_LARGE_STRING, - NANOARROW_TYPE_LARGE_BINARY, + ArrowType.NANOARROW_TYPE_LARGE_STRING, + ArrowType.NANOARROW_TYPE_LARGE_BINARY, ): dtype = (DtypeKind.INT, 64, "l", "=") else: @@ -133,24 +122,26 @@ def dtype(self) -> Dtype: Returns the data type of the column. The returned dtype provides information on the storage format and the type of data in the column. """ - if self.ora_arrow_array.arrow_type == NANOARROW_TYPE_INT64: + arrow_type = self.ora_arrow_array.arrow_type + if arrow_type == ArrowType.NANOARROW_TYPE_INT64: return (DtypeKind.INT, 64, "l", "=") - elif self.ora_arrow_array.arrow_type == NANOARROW_TYPE_DOUBLE: + elif arrow_type == ArrowType.NANOARROW_TYPE_DOUBLE: return (DtypeKind.FLOAT, 64, "g", "=") - elif self.ora_arrow_array.arrow_type == NANOARROW_TYPE_FLOAT: + elif arrow_type == ArrowType.NANOARROW_TYPE_FLOAT: return (DtypeKind.FLOAT, 64, "g", "=") - elif self.ora_arrow_array.arrow_type == NANOARROW_TYPE_STRING: + elif arrow_type == ArrowType.NANOARROW_TYPE_STRING: return (DtypeKind.STRING, 8, "u", "=") - elif self.ora_arrow_array.arrow_type == NANOARROW_TYPE_TIMESTAMP: - if self.ora_arrow_array.time_unit == NANOARROW_TIME_UNIT_MICRO: + elif arrow_type == ArrowType.NANOARROW_TYPE_TIMESTAMP: + time_unit = self.ora_arrow_array.time_unit + if time_unit == ArrowTimeUnit.NANOARROW_TIME_UNIT_MICRO: return (DtypeKind.DATETIME, 64, "tsu:", "=") - elif self.ora_arrow_array.time_unit == NANOARROW_TIME_UNIT_SECOND: + elif time_unit == ArrowTimeUnit.NANOARROW_TIME_UNIT_SECOND: return (DtypeKind.DATETIME, 64, "tss:", "=") - elif self.ora_arrow_array.time_unit == NANOARROW_TIME_UNIT_MILLI: + elif time_unit == ArrowTimeUnit.NANOARROW_TIME_UNIT_MILLI: return (DtypeKind.DATETIME, 64, "tsm:", "=") - elif self.ora_arrow_array.time_unit == NANOARROW_TIME_UNIT_NANO: + elif time_unit == ArrowTimeUnit.NANOARROW_TIME_UNIT_NANO: return (DtypeKind.DATETIME, 64, "tsn:", "=") - elif self.ora_arrow_array.arrow_type == NANOARROW_TYPE_DECIMAL128: + elif arrow_type == ArrowType.NANOARROW_TYPE_DECIMAL128: array = self.ora_arrow_array return ( DtypeKind.DECIMAL, @@ -158,11 +149,11 @@ def dtype(self) -> Dtype: f"d:{array.precision}.{array.scale}", "=", ) - elif self.ora_arrow_array.arrow_type == NANOARROW_TYPE_BINARY: + elif arrow_type == ArrowType.NANOARROW_TYPE_BINARY: return (DtypeKind.STRING, 8, "z", "=") - elif self.ora_arrow_array.arrow_type == NANOARROW_TYPE_LARGE_BINARY: + elif arrow_type == ArrowType.NANOARROW_TYPE_LARGE_BINARY: return (DtypeKind.STRING, 8, "Z", "=") - elif self.ora_arrow_array.arrow_type == NANOARROW_TYPE_LARGE_STRING: + elif arrow_type == ArrowType.NANOARROW_TYPE_LARGE_STRING: return (DtypeKind.STRING, 8, "U", "=") def get_buffers(self) -> ColumnBuffers: diff --git a/src/oracledb/interchange/nanoarrow_bridge.pxd b/src/oracledb/interchange/nanoarrow_bridge.pxd index 479fa7d0..5c413f22 100644 --- a/src/oracledb/interchange/nanoarrow_bridge.pxd +++ b/src/oracledb/interchange/nanoarrow_bridge.pxd @@ -41,6 +41,9 @@ cdef extern from "nanoarrow.h": int64_t null_count int64_t offset int64_t n_buffers + int64_t n_children + ArrowArray** children + const void** buffers void (*release)(ArrowArray*) cdef struct ArrowSchema: @@ -57,6 +60,7 @@ cdef extern from "nanoarrow.h": NANOARROW_TYPE_LARGE_STRING NANOARROW_TYPE_STRING NANOARROW_TYPE_TIMESTAMP + NANOARROW_TYPE_UNINITIALIZED cpdef enum ArrowTimeUnit: NANOARROW_TIME_UNIT_SECOND @@ -87,7 +91,6 @@ cdef class OracleArrowArray: double factor ArrowArray *arrow_array ArrowSchema *arrow_schema - void (*actual_array_release)(ArrowArray*) noexcept cdef str _schema_to_string(self) cdef int append_bytes(self, void* ptr, int64_t num_bytes) except -1 diff --git a/src/oracledb/interchange/nanoarrow_bridge.pyx b/src/oracledb/interchange/nanoarrow_bridge.pyx index dd931d8c..34fc6c0d 100644 --- a/src/oracledb/interchange/nanoarrow_bridge.pyx +++ b/src/oracledb/interchange/nanoarrow_bridge.pyx @@ -31,7 +31,6 @@ cimport cpython from libc.stdint cimport uintptr_t from libc.string cimport memcpy, strlen, strchr -from cpython.pycapsule cimport PyCapsule_New from .. import errors @@ -39,6 +38,16 @@ cdef extern from "nanoarrow/nanoarrow.c": ctypedef int ArrowErrorCode + ctypedef void (*ArrowBufferDeallocatorCallback) + + cdef struct ArrowBufferAllocator: + void *private_data + + cdef struct ArrowBuffer: + uint8_t *data + int64_t size_bytes + ArrowBufferAllocator allocator + cdef union ArrowBufferViewData: const void* data @@ -49,10 +58,6 @@ cdef extern from "nanoarrow/nanoarrow.c": cdef struct ArrowArrayView: ArrowBufferView *buffer_views - cdef struct ArrowBuffer: - uint8_t *data - int64_t size_bytes - cdef struct ArrowDecimal: pass @@ -65,21 +70,21 @@ cdef extern from "nanoarrow/nanoarrow.c": cdef ArrowErrorCode NANOARROW_OK - void ArrowArrayRelease(ArrowArray *array) - void ArrowSchemaRelease(ArrowSchema *schema) - - ArrowErrorCode ArrowArrayInitFromType(ArrowArray* array, - ArrowType storage_type) + ArrowErrorCode ArrowArrayAllocateChildren(ArrowArray *array, + int64_t n_children) ArrowErrorCode ArrowArrayAppendBytes(ArrowArray* array, ArrowBufferView value) - ArrowErrorCode ArrowArrayAppendDouble(ArrowArray* array, double value) - ArrowErrorCode ArrowArrayAppendNull(ArrowArray* array, int64_t n) - ArrowErrorCode ArrowArrayAppendInt(ArrowArray* array, int64_t value) ArrowErrorCode ArrowArrayAppendDecimal(ArrowArray* array, const ArrowDecimal* value) + ArrowErrorCode ArrowArrayAppendDouble(ArrowArray* array, double value) + ArrowErrorCode ArrowArrayAppendInt(ArrowArray* array, int64_t value) + ArrowErrorCode ArrowArrayAppendNull(ArrowArray* array, int64_t n) ArrowBuffer* ArrowArrayBuffer(ArrowArray* array, int64_t i) ArrowErrorCode ArrowArrayFinishBuildingDefault(ArrowArray* array, ArrowError* error) + ArrowErrorCode ArrowArrayInitFromType(ArrowArray* array, + ArrowType storage_type) + void ArrowArrayRelease(ArrowArray *array) ArrowErrorCode ArrowArrayReserve(ArrowArray* array, int64_t additional_size_elements) ArrowErrorCode ArrowArrayStartAppending(ArrowArray* array) @@ -90,8 +95,19 @@ cdef extern from "nanoarrow/nanoarrow.c": const ArrowArray* array, ArrowError* error) int8_t ArrowBitGet(const uint8_t* bits, int64_t i) + ArrowBufferAllocator ArrowBufferDeallocator(ArrowBufferDeallocatorCallback, + void *private_data) + void ArrowDecimalInit(ArrowDecimal* decimal, int32_t bitwidth, + int32_t precision, int32_t scale) + void ArrowDecimalSetBytes(ArrowDecimal *decimal, const uint8_t* value) + ArrowErrorCode ArrowDecimalSetDigits(ArrowDecimal* decimal, + ArrowStringView value) + ArrowErrorCode ArrowSchemaDeepCopy(const ArrowSchema *schema, + ArrowSchema *schema_out) void ArrowSchemaInit(ArrowSchema* schema) ArrowErrorCode ArrowSchemaInitFromType(ArrowSchema* schema, ArrowType type) + void ArrowSchemaRelease(ArrowSchema *schema) + ArrowErrorCode ArrowSchemaSetName(ArrowSchema* schema, const char* name) ArrowErrorCode ArrowSchemaSetTypeDateTime(ArrowSchema* schema, ArrowType arrow_type, ArrowTimeUnit time_unit, @@ -100,15 +116,8 @@ cdef extern from "nanoarrow/nanoarrow.c": ArrowType type, int32_t decimal_precision, int32_t decimal_scale) - ArrowErrorCode ArrowSchemaSetName(ArrowSchema* schema, const char* name) int64_t ArrowSchemaToString(const ArrowSchema* schema, char* out, int64_t n, char recursive) - void ArrowDecimalInit(ArrowDecimal* decimal, int32_t bitwidth, - int32_t precision, int32_t scale) - void ArrowDecimalSetBytes(ArrowDecimal *decimal, const uint8_t* value) - ArrowErrorCode ArrowDecimalSetDigits(ArrowDecimal* decimal, - ArrowStringView value) - cdef int _check_nanoarrow(int code) except -1: """ @@ -119,22 +128,13 @@ cdef int _check_nanoarrow(int code) except -1: errors._raise_err(errors.ERR_ARROW_C_API_ERROR, code=code) -cdef void array_deleter(ArrowArray *array) noexcept: - """ - Called when an external library calls the release for an Arrow array. This - method simply marks the release as completed but doesn't actually do it, so - that the handling of duplicate rows can still make use of the array, even - if the external library no longer requires it! - """ - array.release = NULL - - cdef void pycapsule_array_deleter(object array_capsule) noexcept: cdef ArrowArray* array = cpython.PyCapsule_GetPointer( array_capsule, "arrow_array" ) if array.release != NULL: ArrowArrayRelease(array) + cpython.PyMem_Free(array) cdef void pycapsule_schema_deleter(object schema_capsule) noexcept: @@ -143,6 +143,65 @@ cdef void pycapsule_schema_deleter(object schema_capsule) noexcept: ) if schema.release != NULL: ArrowSchemaRelease(schema) + cpython.PyMem_Free(schema) + + +cdef void arrow_buffer_dealloc_callback(ArrowBufferAllocator *allocator, + uint8_t *ptr, int64_t size): + """ + ArrowBufferDeallocatorCallback for an ArrowBuffer borrowed from + OracleArrowArray + """ + cpython.Py_DECREF( allocator.private_data) + + +cdef int copy_arrow_array(OracleArrowArray oracle_arrow_array, + ArrowArray *src, ArrowArray *dest) except -1: + """ + Shallow copy source ArrowArray to destination ArrowArray. The source + ArrowArray belongs to the wrapper OracleArrowArray. The shallow copy idea + is borrowed from nanoarrow: + https://github.com/apache/arrow-nanoarrow/main/blob/python + """ + cdef: + ArrowBuffer *dest_buffer + ssize_t i + _check_nanoarrow( + ArrowArrayInitFromType( + dest, NANOARROW_TYPE_UNINITIALIZED + ) + ) + + # Copy metadata + dest.length = src.length + dest.offset = src.offset + dest.null_count = src.null_count + + # Borrow an ArrowBuffer belonging to OracleArrowArray. The ArrowBuffer can + # belong to an immediate ArrowArray or a child (in case of nested types). + # Either way, we PY_INCREF(oracle_arrow_array), so that it is not + # prematurely garbage collected. The corresponding PY_DECREF happens in the + # ArrowBufferDeAllocator callback. + for i in range(src.n_buffers): + if src.buffers[i] != NULL: + dest_buffer = ArrowArrayBuffer(dest, i) + dest_buffer.data = src.buffers[i] + dest_buffer.size_bytes = 0 + dest_buffer.allocator = ArrowBufferDeallocator( + arrow_buffer_dealloc_callback, + oracle_arrow_array + ) + cpython.Py_INCREF(oracle_arrow_array) + dest.buffers[i] = src.buffers[i] + dest.n_buffers = src.n_buffers + + # shallow copy of children (recursive call) + if src.n_children > 0: + _check_nanoarrow(ArrowArrayAllocateChildren(dest, src.n_children)) + for i in range(src.n_children): + copy_arrow_array( + oracle_arrow_array, src.children[i], dest.children[i] + ) cdef class OracleArrowArray: @@ -189,8 +248,6 @@ cdef class OracleArrowArray: def __dealloc__(self): if self.arrow_array != NULL: - if self.arrow_array.release == NULL: - self.arrow_array.release = self.actual_array_release if self.arrow_array.release != NULL: ArrowArrayRelease(self.arrow_array) cpython.PyMem_Free(self.arrow_array) @@ -411,6 +468,26 @@ cdef class OracleArrowArray: def offset(self) -> int: return self.arrow_array.offset + def __arrow_c_schema__(self): + """ + Export an ArrowSchema PyCapsule + """ + cdef ArrowSchema *exported_schema = \ + cpython.PyMem_Malloc(sizeof(ArrowSchema)) + try: + _check_nanoarrow( + ArrowSchemaDeepCopy( + self.arrow_schema, + exported_schema + ) + ) + except: + cpython.PyMem_Free(exported_schema) + raise + return cpython.PyCapsule_New( + exported_schema, 'arrow_schema', &pycapsule_schema_deleter + ) + def __arrow_c_array__(self, requested_schema=None): """ Returns @@ -421,13 +498,14 @@ cdef class OracleArrowArray: """ if requested_schema is not None: raise NotImplementedError("requested_schema") - - array_capsule = PyCapsule_New( - self.arrow_array, 'arrow_array', &pycapsule_array_deleter - ) - self.actual_array_release = self.arrow_array.release - self.arrow_array.release = array_deleter - schema_capsule = PyCapsule_New( - self.arrow_schema, "arrow_schema", &pycapsule_schema_deleter - ) - return schema_capsule, array_capsule + cdef ArrowArray *exported_array = \ + cpython.PyMem_Malloc(sizeof(ArrowArray)) + try: + copy_arrow_array(self, self.arrow_array, exported_array) + array_capsule = cpython.PyCapsule_New( + exported_array, 'arrow_array', &pycapsule_array_deleter + ) + except: + cpython.PyMem_Free(exported_array) + raise + return self.__arrow_c_schema__(), array_capsule diff --git a/src/oracledb/plugins/azure_config_provider.py b/src/oracledb/plugins/azure_config_provider.py index c7cb7ca2..52719f85 100644 --- a/src/oracledb/plugins/azure_config_provider.py +++ b/src/oracledb/plugins/azure_config_provider.py @@ -188,9 +188,10 @@ def _parse_parameters(protocol_arg: str) -> dict: parameters = { key.lower(): value[0] for key, value in parsed_values.items() } - parameters["appconfigname"] = ( - protocol_arg[:pos].rstrip("/").rstrip(".azconfig.io") + ".azconfig.io" - ) + config_name = protocol_arg[:pos].rstrip("/") + if not config_name.endswith(".azconfig.io"): + config_name += ".azconfig.io" + parameters["appconfigname"] = config_name return parameters diff --git a/src/oracledb/version.py b/src/oracledb/version.py index ad72b1ea..402c8f44 100644 --- a/src/oracledb/version.py +++ b/src/oracledb/version.py @@ -30,4 +30,4 @@ # file doc/src/conf.py both reference this file directly. # ----------------------------------------------------------------------------- -__version__ = "3.1.0" +__version__ = "3.1.1" diff --git a/tests/test_7700_sparse_vector.py b/tests/test_7700_sparse_vector.py index 9f008713..56cd8a12 100644 --- a/tests/test_7700_sparse_vector.py +++ b/tests/test_7700_sparse_vector.py @@ -222,7 +222,7 @@ def test_7709(self): value = oracledb.SparseVector( 16, [1, 3, 5], array.array("d", [1.5, 0.25, 0.5]) ) - self.__test_insert_and_fetch(value, "VectorFlexAllCol", "f") + self.__test_insert_and_fetch(value, "VectorFlexAllCol", "d") self.__test_insert_and_fetch_sparse( value, "SparseVectorFlexAllCol", "d" ) @@ -357,7 +357,7 @@ def test_7715(self): value = oracledb.SparseVector( 16, [1, 3, 5], array.array("b", [1, 0, 5]) ) - self.__test_insert_and_fetch(value, "VectorFlexAllCol", "f") + self.__test_insert_and_fetch(value, "VectorFlexAllCol", "b") self.__test_insert_and_fetch_sparse( value, "SparseVectorFlexAllCol", "b" ) @@ -442,7 +442,7 @@ def test_7722(self): dim, [1, 3, 5], array.array(typ, [element_value] * 3) ) self.__test_insert_and_fetch( - value, "VectorFlexAllCol", "f" + value, "VectorFlexAllCol", typ ) self.__test_insert_and_fetch_sparse( value, "SparseVectorFlexAllCol", typ @@ -682,9 +682,9 @@ def test_7734(self): self.assertEqual(value.values, array.array("d")) self.assertEqual(value.indices, array.array("I")) self.assertEqual(value.num_dimensions, 0) - with self.assertRaisesFullCode("ORA-51803", "ORA-21560"): + with self.assertRaisesFullCode("ORA-51803", "ORA-21560", "ORA-51862"): self.__test_insert_and_fetch(value, "VectorFlexAllCol", "d") - with self.assertRaisesFullCode("ORA-51803", "ORA-21560"): + with self.assertRaisesFullCode("ORA-51803", "ORA-21560", "ORA-51862"): self.__test_insert_and_fetch_sparse( value, "SparseVectorFlexAllCol", "d" ) diff --git a/tests/test_8000_dataframe.py b/tests/test_8000_dataframe.py index d287b249..3abb84ec 100644 --- a/tests/test_8000_dataframe.py +++ b/tests/test_8000_dataframe.py @@ -409,18 +409,13 @@ def test_8009(self): self.__test_df_batches_interop(DATASET_4, batch_size=5, num_batches=2) def test_8010(self): - "8010 - verify passing Arrow arrays twice fails" + "8010 - verify passing Arrow arrays twice works" self.__check_interop() self.__populate_table(DATASET_1) statement = "select * from TestDataFrame order by Id" ora_df = self.conn.fetch_df_all(statement) - pyarrow.Table.from_arrays( - ora_df.column_arrays(), names=ora_df.column_names() - ) - with self.assertRaises(pyarrow.lib.ArrowInvalid): - pyarrow.Table.from_arrays( - ora_df.column_arrays(), names=ora_df.column_names() - ) + self.__validate_df(ora_df, DATASET_1) + self.__validate_df(ora_df, DATASET_1) def test_8011(self): "8011 - verify empty data set" diff --git a/tests/test_8100_dataframe_async.py b/tests/test_8100_dataframe_async.py index 5cebcbd0..5d9c3de1 100644 --- a/tests/test_8100_dataframe_async.py +++ b/tests/test_8100_dataframe_async.py @@ -420,18 +420,13 @@ async def test_8109(self): ) async def test_8110(self): - "8110 - verify passing Arrow arrays twice fails" + "8110 - verify passing Arrow arrays twice works" self.__check_interop() await self.__populate_table(DATASET_1) statement = "select * from TestDataFrame order by Id" ora_df = await self.conn.fetch_df_all(statement) - pyarrow.Table.from_arrays( - ora_df.column_arrays(), names=ora_df.column_names() - ) - with self.assertRaises(pyarrow.lib.ArrowInvalid): - pyarrow.Table.from_arrays( - ora_df.column_arrays(), names=ora_df.column_names() - ) + self.__validate_df(ora_df, DATASET_1) + self.__validate_df(ora_df, DATASET_1) async def test_8111(self): "8111 - verify empty data set"