Tags: csv2db/csv2db
Tags
Version 1.6.1 (#65) * Move version forward Signed-off-by: Gerald Venzl <gerald.venzl@gmail.com> * Provide support to truncate table before load (ER #45) Signed-off-by: Gerald Venzl <gerald.venzl@gmail.com> * Remove redundant CREATE USER statement Signed-off-by: Gerald Venzl <gerald.venzl@gmail.com> * Print global batch size into debug output Signed-off-by: Gerald Venzl <gerald.venzl@gmail.com> * Refactor truncate table, commit for Postgres and SQLServer Signed-off-by: Gerald Venzl <gerald.venzl@gmail.com> * Add more tests, refactor Signed-off-by: Gerald Venzl <gerald.venzl@gmail.com> * Commit TRUNCATE TABLE IMMEDIATE for Db2 Signed-off-by: Gerald Venzl <gerald.venzl@gmail.com> * Clean up tables after all loading tests have been executed Signed-off-by: Gerald Venzl <gerald.venzl@gmail.com> * Clean up enum comparision Signed-off-by: Gerald Venzl <gerald.venzl@gmail.com> * Document tests in CHANGELOG.md Signed-off-by: Gerald Venzl <gerald.venzl@gmail.com> * Add design goals for csv2db. Signed-off-by: gvenzl <gerald.venzl@gmail.com> * Enhance debug messages and online help. Signed-off-by: gvenzl <gerald.venzl@gmail.com> * Remove unused exception variables. Signed-off-by: gvenzl <gerald.venzl@gmail.com> * Add new line at the end of file. Signed-off-by: gvenzl <gerald.venzl@gmail.com> * Add test for invalid file type. Signed-off-by: gvenzl <gerald.venzl@gmail.com> * Add ignore errors (#43) and log errors (#46) functionality. Signed-off-by: gvenzl <gerald.venzl@gmail.com> * Only remove additional values if ignore errors (and not log errors) has been set. Signed-off-by: gvenzl <gerald.venzl@gmail.com> * Read all files with UTF-8 encoding Signed-off-by: gvenzl <gerald.venzl@gmail.com> * Update help text for --log option Signed-off-by: gvenzl <gerald.venzl@gmail.com> * Update CHANGELOG.md with new features and changes. Signed-off-by: gvenzl <gerald.venzl@gmail.com> * Add test for BadRecordLogger Signed-off-by: gvenzl <gerald.venzl@gmail.com> * Refactor test data folder structure. New test for insufficient header columns. Signed-off-by: gvenzl <gerald.venzl@gmail.com> * if not needed, function already check it itself Signed-off-by: gvenzl <gerald.venzl@gmail.com> * Explicitly close file descriptor for bad file. Close DB cursor and start a new one in case of errors. Signed-off-by: gvenzl <gerald.venzl@gmail.com> * Update data types for STAGING table. Signed-off-by: gvenzl <gerald.venzl@gmail.com> * Add tests for ignore errors and log errors. Signed-off-by: gvenzl <gerald.venzl@gmail.com> * Update .gitignore for Vagrant Signed-off-by: gvenzl <gerald.venzl@gmail.com> * Move test env files under test_env Signed-off-by: gvenzl <gerald.venzl@gmail.com> * Introduce GitHub Actions for functional tests Signed-off-by: gvenzl <gerald.venzl@gmail.com> * Rename test files to adhere to Python unittest naming scheme Signed-off-by: gvenzl <gerald.venzl@gmail.com> * Correct test file locations in negative functional tests Signed-off-by: gvenzl <gerald.venzl@gmail.com> * Increase verbosity for unit tests Signed-off-by: gvenzl <gerald.venzl@gmail.com> * Correct test file locations in load tests Signed-off-by: gvenzl <gerald.venzl@gmail.com> * Update functional tests Signed-off-by: gvenzl <gerald.venzl@gmail.com> * Rename functional tests file. * Rename functional tests file. Signed-off-by: gvenzl <gerald.venzl@gmail.com> * introduce MySQL GH actions * Change to oracledb driver Signed-off-by: gvenzl <gerald.venzl@gmail.com> * Change tests for different tests per database Signed-off-by: gvenzl <gerald.venzl@gmail.com> * Introducing Oracle GitHub Actions tests. Signed-off-by: gvenzl <gerald.venzl@gmail.com> * Add main method for Oracle unit tests. Signed-off-by: gvenzl <gerald.venzl@gmail.com> * Adjust sleep time Signed-off-by: gvenzl <gerald.venzl@gmail.com> * Setup scripts for Oracle Signed-off-by: gvenzl <gerald.venzl@gmail.com> * Oracle tests * Raise error when csv2db loading doesn't succeed. Signed-off-by: gvenzl <gerald.venzl@gmail.com> * Wait for Oracle DB container to be ready Signed-off-by: gvenzl <gerald.venzl@gmail.com> * Move test infrastructure files Signed-off-by: gvenzl <gerald.venzl@gmail.com> * Modularize tests for Oracle, MySQL, Postgres, SQL Server Signed-off-by: gvenzl <gerald.venzl@gmail.com> * Update test infrastructure files. Signed-off-by: gvenzl <gerald.venzl@gmail.com> * Remove old test infrastructure drop file (now using GH Actions) Signed-off-by: gvenzl <gerald.venzl@gmail.com> * Removing old Python3 versions from test matrix Signed-off-by: gvenzl <gerald.venzl@gmail.com> * Removing old Python3 versions from functional test matrix Signed-off-by: gvenzl <gerald.venzl@gmail.com> * Introducing Postgres GHA tests. Signed-off-by: gvenzl <gerald.venzl@gmail.com> * Update Postgres GHA file Signed-off-by: gvenzl <gerald.venzl@gmail.com> * Update Postgres GHA file Signed-off-by: gvenzl <gerald.venzl@gmail.com> * Pass on correct DBType for truncate Signed-off-by: gvenzl <gerald.venzl@gmail.com> * Rename loading suite Signed-off-by: gvenzl <gerald.venzl@gmail.com> * Commit transactions for Postgres. Postgres raises "current transaction is aborted, commands ignored until end of transaction block" when an error is encountered. Subsequent SQL commands all will receive this error until the transaction is ended. COMMIT doesn't appear to be a reliable way to ensure previous successful rows are committed, hence in an error case every row is committed. Signed-off-by: gvenzl <gerald.venzl@gmail.com> * Add debug output for load test. Use global variables for DATABASE_ERROR test. Signed-off-by: gvenzl <gerald.venzl@gmail.com> * Rewrite batch error handling: * SQL Server and Postgres cannot continue a transaction once it had an error. * Add counters for successful and ignored errors (when ignore error flag is on) Signed-off-by: gvenzl <gerald.venzl@gmail.com> * Update Postgres Tests Signed-off-by: gvenzl <gerald.venzl@gmail.com> * Use GitHub Action for Oracle DB Free * Introduce SQL Server GHA tests Signed-off-by: gvenzl <gerald.venzl@gmail.com> * Introduce SQL Server GHA tests Signed-off-by: gvenzl <gerald.venzl@gmail.com> * Create SQL Server test database Signed-off-by: gvenzl <gerald.venzl@gmail.com> * Create SQL Server test database Signed-off-by: gvenzl <gerald.venzl@gmail.com> * Create SQL Server test database Signed-off-by: gvenzl <gerald.venzl@gmail.com> * Create SQL Server test database Signed-off-by: gvenzl <gerald.venzl@gmail.com> * Use Oracle slim image Signed-off-by: gvenzl <gerald.venzl@gmail.com> * Increase health interval to have database user created Signed-off-by: gvenzl <gerald.venzl@gmail.com> * Increase health interval to have database user created Signed-off-by: gvenzl <gerald.venzl@gmail.com> * Restructure test folders. Signed-off-by: gvenzl <gerald.venzl@gmail.com> * Fix postgres and mysql tests Signed-off-by: gvenzl <gerald.venzl@gmail.com> * Connect through localhost Signed-off-by: gvenzl <gerald.venzl@gmail.com> * Fix MySQL tests Signed-off-by: gvenzl <gerald.venzl@gmail.com> * Fixing MySQL tests. Signed-off-by: gvenzl <gerald.venzl@gmail.com> * Fix MySQL tests. Signed-off-by: gvenzl <gerald.venzl@gmail.com> * Intro DB2 GHA tests. Signed-off-by: gvenzl <gerald.venzl@gmail.com> * Include longer pause in MySQL Tests. Signed-off-by: gvenzl <gerald.venzl@gmail.com> * Update Db2 tests. Signed-off-by: gvenzl <gerald.venzl@gmail.com> * Fix MySQL tests with init.d schema setup. Signed-off-by: gvenzl <gerald.venzl@gmail.com> * Add Db2 load tests Signed-off-by: gvenzl <gerald.venzl@gmail.com> * Rename Db2 test files Signed-off-by: gvenzl <gerald.venzl@gmail.com> * Correct schema file path Signed-off-by: gvenzl <gerald.venzl@gmail.com> * Upgrade to `psycopg 3` Signed-off-by: gvenzl <gerald.venzl@gmail.com> * Scripts for local Podman/Docker setup Signed-off-by: gvenzl <gerald.venzl@gmail.com> * Fix Postgres GHA Signed-off-by: gvenzl <gerald.venzl@gmail.com> * Specify encoding for zip file entries Signed-off-by: gvenzl <gerald.venzl@gmail.com> * Provide file encoding parameter (#50) Signed-off-by: gvenzl <gerald.venzl@gmail.com> * Update Python 3 supported version testing matrix Signed-off-by: gvenzl <gerald.venzl@gmail.com> * Update testing password Signed-off-by: gvenzl <gerald.venzl@gmail.com> * Update test password to make SQL Server work again Signed-off-by: gvenzl <gerald.venzl@gmail.com> * Update CHANGELOG.md Signed-off-by: gvenzl <gerald.venzl@gmail.com> * Fix bug #56 Signed-off-by: gvenzl <gerald.venzl@gmail.com> * Implement ER #60 Signed-off-by: gvenzl <gerald.venzl@gmail.com> * Factor out constants into separate constants.pyn Signed-off-by: gvenzl <gerald.venzl@gmail.com> * Implement #61, enable quoted identifiers. Signed-off-by: gvenzl <gerald.venzl@gmail.com> * Restructure package Signed-off-by: gvenzl <gerald.venzl@gmail.com> * update pyproject.toml Signed-off-by: gvenzl <gerald.venzl@gmail.com> * Create release.yml * Publish to PyPI * Publish to Test PyPI * Fix typos * Fix typo Signed-off-by: gvenzl <gerald.venzl@gmail.com> * Update project test version Signed-off-by: gvenzl <gerald.venzl@gmail.com> * Fix typo Signed-off-by: gvenzl <gerald.venzl@gmail.com> * Introduce PyPI publishing * Release csv2db as Python package (ER #53) Signed-off-by: gvenzl <gerald.venzl@gmail.com> * Remove ibm-db from requirements. ibm-db is still in beta phase and a thick driver (requiring C library versions that are not everywhere easily available) Signed-off-by: gvenzl <gerald.venzl@gmail.com> * Update ReadMe with Python package installation, minor fixes Signed-off-by: gvenzl <gerald.venzl@gmail.com> * Add Python 3.8 and 3.9 tests back in as they are still supported. Signed-off-by: gvenzl <gerald.venzl@gmail.com> * Add Python 3.8 and 3.9 tests back in, remove not yet supported Python 3.12 version. Signed-off-by: gvenzl <gerald.venzl@gmail.com> * Include Python 3.12 version Signed-off-by: gvenzl <gerald.venzl@gmail.com> * Typo in test description Signed-off-by: gvenzl <gerald.venzl@gmail.com> * Sleep of 10 seconds still sometimes triggers test run failure. Signed-off-by: gvenzl <gerald.venzl@gmail.com> * Update Changelog release Signed-off-by: gvenzl <gerald.venzl@gmail.com> * Harmonize GitHub Actions workflow files Signed-off-by: gvenzl <gerald.venzl@gmail.com> * Roll forward CHANGELOG.md Signed-off-by: Gerald Venzl <gerald.venzl@gmail.com> * Fix psycopg[binary] installation Signed-off-by: Gerald Venzl <gerald.venzl@gmail.com> * Open MySQL user to all IP addresses Signed-off-by: Gerald Venzl <gerald.venzl@gmail.com> * Introducing Dockerfile #63 (superseding #48) Signed-off-by: Gerald Venzl <gerald.venzl@gmail.com> * Switch from console to bash as some tools enable automatic run for bash Signed-off-by: Gerald Venzl <gerald.venzl@gmail.com> * Docker publish Signed-off-by: Gerald Venzl <gerald.venzl@gmail.com> * Checkout Docker build Signed-off-by: Gerald Venzl <gerald.venzl@gmail.com> * Add label details for Docker image Signed-off-by: Gerald Venzl <gerald.venzl@gmail.com> * Remove incomplete workflow Signed-off-by: Gerald Venzl <gerald.venzl@gmail.com> * Introduce Registries Publishing workflows Signed-off-by: Gerald Venzl <gerald.venzl@gmail.com> * Update Docker image publishing workflow (#64) Adds new automated publishing of Docker images onto both, ` ghcr.io/csv2db/csv2db` and `docker.io/gvenzl/csv2db`, finalizing #63 * Update oracledb to 2.0.0+ Signed-off-by: Gerald Venzl <gerald.venzl@gmail.com> * Move to v1.6.1 Signed-off-by: Gerald Venzl <gerald.venzl@gmail.com> --------- Signed-off-by: Gerald Venzl <gerald.venzl@gmail.com> Signed-off-by: gvenzl <gerald.venzl@gmail.com>
v1.5.1 (#40) * Introduce .gitattributes Signed-off-by: Gerald Venzl <gerald.venzl@gmail.com> * Remove tabs in .gitattribute Signed-off-by: Gerald Venzl <gerald.venzl@gmail.com> * Add password prompt Signed-off-by: Gerald Venzl <gerald.venzl@gmail.com> * Correct path in unit test test file Signed-off-by: Gerald Venzl <gerald.venzl@gmail.com> * Introduce interactive password Signed-off-by: Gerald Venzl <gerald.venzl@gmail.com> * Move version number forward. Signed-off-by: Gerald Venzl <gerald.venzl@gmail.com> * Introduce tests for different databases Signed-off-by: Gerald Venzl <gerald.venzl@gmail.com> * Include server authentication and protocol in DB2 connect string. Otherwise the driver tries to authenticate against a local user instead of the one on the database server. Signed-off-by: Gerald Venzl <gerald.venzl@gmail.com> * Add unicode tests. Signed-off-by: Gerald Venzl <gerald.venzl@gmail.com> * Rename for new test case Signed-off-by: Gerald Venzl <gerald.venzl@gmail.com> * Add load test for Oracle. Signed-off-by: Gerald Venzl <gerald.venzl@gmail.com> * Convert data list to tuple at array append. Sql Server requires a tuple or dictionary for insert. Rather than converting the line straight to a tuple and lose functionality like list.pop(), conversion is now happening at the very last time that the row values are touched (when appending it to the input_array. Bug fix for bug #35 Signed-off-by: Gerald Venzl <gerald.venzl@gmail.com> * Use UTF-8 connection encoding for Oracle (default: ascii) All other databases use UTF-8 by default. Bug fix for bug #36 Signed-off-by: Gerald Venzl <gerald.venzl@gmail.com> * Enhance error reporting (ER #37) Signed-off-by: Gerald Venzl <gerald.venzl@gmail.com> * Separate functional tests from loading tests. Signed-off-by: Gerald Venzl <gerald.venzl@gmail.com> * Fix text description prints Signed-off-by: Gerald Venzl <gerald.venzl@gmail.com> * Using `psycopg2.extras.execute_batch()` for Postgres (ER #39) Signed-off-by: Gerald Venzl <gerald.venzl@gmail.com> * v.1.5.1 release Signed-off-by: Gerald Venzl <gerald.venzl@gmail.com> * Fix doc string for executemany Signed-off-by: Gerald Venzl <gerald.venzl@gmail.com>
1.5.0 release (#34) * Introduce .gitattributes Signed-off-by: Gerald Venzl <gerald.venzl@gmail.com> * Remove tabs in .gitattribute Signed-off-by: Gerald Venzl <gerald.venzl@gmail.com> * Add password prompt Signed-off-by: Gerald Venzl <gerald.venzl@gmail.com> * Correct path in unit test test file Signed-off-by: Gerald Venzl <gerald.venzl@gmail.com> * Introduce interactive password Signed-off-by: Gerald Venzl <gerald.venzl@gmail.com>
Version 1.2.0 (#7) * Forward changelog Signed-off-by: Gerald Venzl <gerald.venzl@gmail.com> * Bugfix (#6): Don't execute batch flush on empty list Signed-off-by: Gerald Venzl <gerald.venzl@gmail.com> * Introduce support for user-defined column separator Signed-off-by: Gerald Venzl <gerald.venzl@gmail.com> * Version 1.2.0 release Signed-off-by: Gerald Venzl <gerald.venzl@gmail.com> * Version 1.2.0 release Signed-off-by: Gerald Venzl <gerald.venzl@gmail.com>
PreviousNext