Replies: 4 comments 4 replies
-
The code looks like that, I wanted to benchmark the use of "executemany" vs "insert all". with connect_to_copied_db(copy_db_info) as copy_connection: # sqlalchemy connection
with src_cnx.engine.raw_connection().cursor() as src_cursor, \
copy_connection.connection.cursor() as copy_cursor:
for table_name, query in filtered_table_queries:
src_cursor.arraysize = table_batch_size(table_name)
try:
src_cursor.execute(query)
except Exception as e:
logger.exception(f'Cannot execute query {query}')
raise e from None
fulltable = f'{copy_db_info.schema}.{table_name}'
previous_nb_rows = 0
while batch := src_cursor.fetchmany():
nb_rows = len(batch)
if nb_rows != previous_nb_rows:
num_columns = len(batch[0])
previous_nb_rows = nb_rows
insert_query = 'insert all '
for irow in range(nb_rows):
placeholders = ', '.join([f":{irow*num_columns + i + 1}" for i in range(num_columns)])
insert_query += f' into {fulltable} values ({", ".join([placeholders])})\n'
insert_query += 'select * from dual'
params = []
for row in batch:
params.extend(list(row))
try:
copy_cursor.execute(insert_query, params)
except Exception as e:
logger.exception(f'Cannot insert into {table_name}')
raise e from None
copy_connection.connection.commit() |
Beta Was this translation helpful? Give feedback.
-
Thanks for the report. Note that in 23ai you can also try a multi-value insert https://github.com/oracle/python-oracledb/blob/v3.0.0/samples/sql/create_schema_23.sql#L62-L70 Regarding the benchmark: using a large number of binds could hit the issue in oracle/python-cx_Oracle#567 Although binds are handled linearly in python-oracledb Thin mode itself, and in Thick mode the Oracle Client 23ai libraries improved its handling of large numbers of them, the DB itself still doesn't give the performance I think it should when a large number of binds are used in a statement - I was chasing this up just last week. Fundamentally, the best recommendation is to use SQL*Loader for loading data. |
Beta Was this translation helpful? Give feedback.
-
Anything is possible! :-) Can you share a test case that reliably demonstrates the problem? A quick glance suggests, however, that you are passing bytes that are not UTF-8 encoded to a string bind variable. I'd be happy to take a look, though, if you can supply a test case. |
Beta Was this translation helpful? Give feedback.
-
@nmoreaud how is your testing going? |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
Hello
I build queries that look like this:
Depending on how many rows I try to insert at once, I get this error:
However, the same insertion with this syntax works well:
The error seems to happen for tables containing blobs.
Do you think there might be a parsing error or a memory issue in oracledb?
Beta Was this translation helpful? Give feedback.
All reactions