Skip to content

Update regression tests

Jump to bottom
Oleg Piskunov edited this page Mar 27, 2020 · 8 revisions

We have a set of regression tests which are difficult to maintain. There are two main problems:

  • using reference output instead of asserts
  • mixing many testcases in a single file

We want to split testcases to an independent functions of files and replace comparing of actual output with reference output by asserts in code. Table below contains our backlog with such regerssion tests and status of updating them.

Tests marked with bold have first priority.

app

Test Comment Status Owner
app/crypto_hmac.test.lua TAP13, easy
app/crypto.test.lua TAP13, easy
app/decimal.test.lua TAP13, easy
app/digest.test.lua TAP13, easy
app/fiber_channel.test.lua
app/fiber.test.lua
app/fio.test.lua
app/loaders.test.lua
app/msgpack.test.lua TAP13, easy
app/pack.test.lua TAP13, easy
app/socket.test.lua TAP13, not easy
app/uuid.test.lua TAP13, easy

box

Test Comment Status Owner
box/access_bin.test.lua TAP13, easy
box/access_escalation.test.lua on review https://github.com/tarantool/tarantool/tree/opiskunov/access_escal_test_split @oleg-piskunov
box/access_misc.test.lua TAP13, easy
box/access_sysview.test.lua TAP13, easy
box/access.test.lua TAP13, easy
box/admin.test.lua TAP13, easy
box/alter_limits.test.lua TAP13
box/backup.test.lua TAP13, easy
box/backup_test.lua TAP13, easy
box/before_replace.test.lua TAP13, easy
box/bitset.test.lua TAP13, easy
box/blackhole.test.lua TAP13
box/call.test.lua TAP13
box/cfg.test.lua TAP13
box/ddl.test.lua LGTM https://github.com/tarantool/tarantool/tree/avtikhon/divide_tests @avtikhon
box/errinj_index.test.lua TAP13?
box/errinj.test.lua TAP13
box/func_reload.test.lua TAP13, easy
box/function1.test.lua TAP13
box/gh-2763-session-credentials-update.test.lua TAP13, easy
box/gh-4511-access-settings-from-any-frontend.test.lua
box/gh-4513-netbox-self-and-connect-interchangeable.test.lua
box/gh-4593-netbox-on_connect-disconnect.test.lua
box/gh-4627-session-use-after-free.test.lua
box/gh-4648-func-load-unload.test.lua
box/gh-4672-min-integer-value-in-serializer.test.lua
box/gh-4770-broken-pairs-for-space-objects.test.lua
box/hash.test.lua LGTM https://github.com/tarantool/tarantool/tree/ligurio/hash_test_split @ligurio
box/indices_any_type.test.lua
box/info.test.lua
box/iproto_stress.test.lua
box/iterator.test.lua
box/leak.test.lua
box/lua.test.lua
box/misc.test.lua
box/net.box.test.lua in progress https://github.com/tarantool/tarantool/tree/avtikhon/divide_tests @avtikhon
box/net_msg_max.test.lua
box/on_replace.test.lua
box/on_shutdown.test.lua
box/protocol.test.lua
box/push.test.lua
box/reconfigure.test.lua
box/role.test.lua
box/rtree_array.test.lua
box/rtree_errinj.test.lua
box/rtree_misc.test.lua
box/rtree_point_r2.test.lua
box/rtree_point.test.lua

engine

Test Comment Status Owner
engine/conflict.test.lua
engine/crossjoin.test.lua
engine/ddl.test.lua
engine/decimal.test.lua
engine/delete.test.lua
engine/errinj_ddl.test.lua
engine/errinj.test.lua
engine/func_index.test.lua
engine/hints.test.lua
engine/indices_any_type.test.lua
engine/insert.test.lua
engine/iterator.test.lua
engine/json.test.lua
engine/lua.test.lua
engine/misc.test.lua
engine/multikey.test.lua
engine/null.test.lua
engine/params.test.lua
engine/recover_drop.test.lua
engine/recover_snapshot.test.lua
engine/recover_snapshot_wal.test.lua
engine/recover_wal.test.lua
engine/replace.test.lua
engine/replica_join.test.lua
engine/savepoint.test.lua
engine/select.test.lua
engine/snapshot.test.lua
engine/transaction.test.lua
engine/tree_min_max_count.test.lua
engine/tree.test.lua
engine/tree_variants.test.lua
engine/truncate.test.lua
engine/tuple.test.lua
engine/update.test.lua
engine/upsert.test.lua

engine_long

Test Comment Status Owner
engine_long/delete_insert.test.lua
engine_long/delete_replace_update.test.lua

replication

Test Comment Status Owner
replication/anon.test.lua
replication/autobootstrap_guest.test.lua
replication/autobootstrap.test.lua
replication/before_replace.test.lua
replication/bootstrap_leader.test.lua
replication/box_set_replication_stress.test.lua
replication/catch.test.lua
replication/consistent.test.lua
replication/ddl.test.lua
replication/errinj.test.lua
replication/force_recovery.test.lua
replication/gc_no_space.test.lua
replication/gc.test.lua
replication/gh-4402-info-errno.test.lua
replication/gh-4605-empty-password.test.lua
replication/gh-4606-admin-creds.test.lua
replication/gh-4729-netbox-group-id.test.lua
replication/gh-4739-vclock-assert.test.lua
replication/hot_standby.test.lua
replication/join_vclock.test.lua
replication/join_without_snap.test.lua
replication/local_spaces.test.lua
replication/long_row_timeout.test.lua
replication/misc.test.lua
replication/once.test.lua
replication/on_replace.test.lua
replication/on_schema_init.test.lua
replication/prune.test.lua
replication/quorum.test.lua
replication/rebootstrap.test.lua
replication/recover_missing_xlog.test.lua
replication/replica_apply_order.test.lua
replication/replica_rejoin.test.lua
replication/replicaset_ro_mostly.test.lua
replication/show_error_on_disconnect.test.lua
replication/skip_conflict_row.test.lua
replication/status.test.lua
replication/sync.test.lua
replication/transaction.test.lua
replication/wal_off.test.lua
replication/wal_rw_stress.test.lua

sql

Test Comment Status Owner
sql/autoincrement.test.lua
sql/bind.test.lua
sql/check-clear-ephemeral.test.lua
sql/checks.test.lua
sql/clear.test.lua
sql/collation.test.lua
sql/constraint.test.lua
sql/ddl.test.lua
sql/delete-multiple-idx.test.lua
sql/delete.test.lua
sql/drop-index.test.lua
sql/drop-table.test.lua
sql/engine.test.lua
sql/errinj.test.lua
sql/foreign-keys.test.lua
sql/full_metadata.test.lua
sql/func-recreate.test.lua
sql/gh2141-delete-trigger-drop-table.test.lua
sql/gh2251-multiple-update.test.lua
sql/gh-2362-select-access-rights.test.lua
sql/gh2483-remote-persistency-check.test.lua
sql/gh2808-inline-unique-persistency-check.test.lua
sql/gh-2929-primary-key.test.lua
sql/gh-2981-check-autoinc.test.lua
sql/gh-3199-no-mem-leaks.test.lua
sql/gh-3613-idx-alter-update-2.test.lua
sql/gh-3613-idx-alter-update.test.lua
sql/gh-3888-values-blob-assert.test.lua
sql/gh-4104-view-access-check.test.lua
sql/gh-4111-format-in-sysview.test.lua
sql/gh-4546-sql-drop-grants.test.lua
sql/gh-4745-table-info-assertion.test.lua
sql/icu-upper-lower.test.lua
sql/insert-unique.test.lua
sql/integer-overflow.test.lua
sql/iproto.test.lua
sql/max-on-index.test.lua
sql/message-func-indexes.test.lua
sql/misc.test.lua
sql/no-pk-space.test.lua
sql/on-conflict.test.lua
sql/persistency.test.lua
sql/prepared.test.lua
sql/row-count.test.lua
sql/savepoints.test.lua
sql/select-null.test.lua
sql/sql-statN-index-drop.test.lua
sql/tokenizer.test.lua
sql/transition.test.lua
sql/transitive-transactions.test.lua
sql/triggers.test.lua
sql/types.test.lua
sql/update-with-nested-select.test.lua
sql/upgrade.test.lua
sql/view_delayed_wal.test.lua
sql/view.test.lua
sql/vinyl-opts.test.lua

swim

Test Comment Status Owner
swim/swim.test.lua TAP13, easy
swim/errinj.test.lua TAP13

vinyl

Test Comment Status Owner
vinyl/bloom.test.lua
vinyl/cache.test.lua
vinyl/compact.test.lua
vinyl/constraint.test.lua
vinyl/ddl.test.lua
vinyl/deferred_delete.test.lua
vinyl/dump_stress.test.lua
vinyl/errinj_ddl.test.lua
vinyl/errinj_gc.test.lua
vinyl/errinj_stat.test.lua
vinyl/errinj.test.lua
vinyl/errinj_tx.test.lua
vinyl/errinj_vylog.test.lua
vinyl/gc.test.lua
vinyl/gh.test.lua
vinyl/hermitage.test.lua
vinyl/iterator.test.lua
vinyl/json.test.lua
vinyl/large.test.lua
vinyl/layout.test.lua
vinyl/misc.test.lua
vinyl/mvcc.test.lua
vinyl/on_replace.test.lua
vinyl/options.test.lua
vinyl/parallel.test.lua
vinyl/partial_dump.test.lua
vinyl/quota.test.lua
vinyl/quota_timeout.test.lua
vinyl/recover.test.lua
vinyl/recovery_quota.test.lua
vinyl/replica_quota.test.lua
vinyl/replica_rejoin.test.lua
vinyl/savepoint.test.lua
vinyl/select_consistency.test.lua
vinyl/snap_io_rate.test.lua
vinyl/snapshot.test.lua
vinyl/split_coalesce.test.lua
vinyl/stat.test.lua
vinyl/stress.test.lua
vinyl/throttle.test.lua
vinyl/tx_conflict.test.lua
vinyl/tx_gap_lock.test.lua
vinyl/tx_serial.test.lua
vinyl/update_optimize.test.lua
vinyl/upgrade.test.lua
vinyl/upsert.test.lua
vinyl/write_iterator_rand.test.lua
vinyl/write_iterator.test.lua

wal_off

Test Comment Status Owner
wal_off/alter.test.lua
wal_off/expirationd.test.lua
wal_off/func_max.test.lua
wal_off/iterator_lt_gt.test.lua
wal_off/rtree_benchmark.test.lua
wal_off/snapshot_stress.test.lua
wal_off/tuple.test.lua
wal_off/wal_mode.test.lua

xlog

Test Comment Status Owner
xlog/big_tx.test.lua
xlog/checkpoint_daemon.test.lua
xlog/checkpoint_threshold.test.lua
xlog/errinj.test.lua
xlog/force_recovery.test.lua
xlog/gh1433.test.lua
xlog/gh-4771-upgrade.test.lua
xlog/header.test.lua
xlog/misc.test.lua
xlog/panic_on_broken_lsn.test.lua
xlog/panic_on_wal_error.test.lua
xlog/reader.test.lua
xlog/snap_io_rate.test.lua
xlog/transaction.test.lua

Developer Guidelines ↗

Architecture

How To ...?

Recipes

Upgrade instructions

Useful links

Old discussions

Personal pages

Clone this wiki locally