Skip to content

Updating of CI-tests #197

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
47 commits
Select commit Hold shift + click to select a range
7d8302f
Test dockerfile for ubuntu 24.04 is added
dmitry-lipetsk Feb 26, 2025
2df2fd4
Cleanup (Dockerfile--ubuntu-24_04)
dmitry-lipetsk Feb 26, 2025
e8bf7d6
Cleanup.
dmitry-lipetsk Feb 27, 2025
103bcd2
[run_tests.sh] A right way for obtaining of BINDIR and PG_CONFIG is …
dmitry-lipetsk Feb 27, 2025
d0eb414
Merge remote-tracking branch 'origin/D20250227_001--pg_config-bindir'…
dmitry-lipetsk Feb 27, 2025
579d20e
Dockerfile--ubuntu-24_04 is updated
dmitry-lipetsk Feb 27, 2025
e87c402
Dockerfile--ubuntu-24_04 is updated (cleanup)
dmitry-lipetsk Feb 27, 2025
b88327d
Merge commit '9911ed0c928b602c17a3b0845b24910bf291925a' into D2025022…
dmitry-lipetsk Feb 27, 2025
126b5a8
Merge commit '6d67da2170becb944fc768ff4938f88b37e0f2a4' into D2025022…
dmitry-lipetsk Feb 27, 2025
96227f5
Dockerfile--ubuntu-24_04 is updated (cleanup)
dmitry-lipetsk Feb 27, 2025
b7bf713
Dockerfile--ubuntu-24_04 is formatted
dmitry-lipetsk Feb 27, 2025
bcb7771
CI-test on Ubuntu 24.04 is added.
dmitry-lipetsk Feb 27, 2025
7d871a2
Dockerfile--std.tmpl is updated (refactoring)
dmitry-lipetsk Feb 27, 2025
381d64d
Dockerfile--ubuntu-24_04.tmpl is updated
dmitry-lipetsk Feb 27, 2025
b5c4d7b
PostgresNode::pid is improved
dmitry-lipetsk Feb 27, 2025
fce595a
PostgresNode::pid is updated
dmitry-lipetsk Feb 27, 2025
a92a6e7
Merge remote-tracking branch 'origin/D20250227_001--node-pid' into D2…
dmitry-lipetsk Feb 27, 2025
0aebc00
Merge branch 'master' into D20250227_001--node-pid
dmitry-lipetsk Feb 28, 2025
ebd6208
Merge branch 'master' into D20250226_001--ci_and_ssh
dmitry-lipetsk Feb 28, 2025
0863cf5
execute_utility2 is updated (ignore_errors)
dmitry-lipetsk Feb 28, 2025
a4e65bf
Merge branch 'D20250228_001--execute_utility2' into D20250227_001--no…
dmitry-lipetsk Feb 28, 2025
09976ae
PostgresNode::_try_shutdown is rewrited (normalization)
dmitry-lipetsk Feb 28, 2025
ab28c2e
Merge branch '20250228_002--try_shutdown' into D20250227_001--node-pid
dmitry-lipetsk Feb 28, 2025
0402c4a
PostgresNode::pid uses the data from "pg_ctl status" output.
dmitry-lipetsk Feb 28, 2025
c288af8
Merge remote-tracking branch 'origin/D20250227_001--node-pid' into D2…
dmitry-lipetsk Feb 28, 2025
45d2b17
PostgresNode::_try_shutdown is correct (return None)
dmitry-lipetsk Feb 28, 2025
a960ac7
Merge branch '20250228_002--try_shutdown' into D20250227_001--node-pid
dmitry-lipetsk Feb 28, 2025
5c6548f
Merge remote-tracking branch 'origin/D20250227_001--node-pid' into D2…
dmitry-lipetsk Feb 28, 2025
619fefa
[RemoteOperations] A call of mktemp is fixed
dmitry-lipetsk Feb 28, 2025
015b766
Merge remote-tracking branch 'origin/D20250228_001--remote-mktemp' in…
dmitry-lipetsk Feb 28, 2025
82981f5
Merge branch 'master' into D20250227_001--node-pid
dmitry-lipetsk Mar 1, 2025
61d8929
Merge remote-tracking branch 'origin/D20250227_001--node-pid' into D2…
dmitry-lipetsk Mar 1, 2025
3c2b51f
Merge branch 'master' into D20250226_001--ci_and_ssh
dmitry-lipetsk Mar 1, 2025
76e9c53
Merge branch 'master' into D20250226_001--ci_and_ssh
dmitry-lipetsk Mar 1, 2025
1309442
Total refactoring of os_ops::execute_command
dmitry-lipetsk Mar 1, 2025
9b7c10e
Merge remote-tracking branch 'origin/D20250301_01--os_ops-exec_comman…
dmitry-lipetsk Mar 1, 2025
e245e23
Dockerfile--ubuntu-24_04.tmpl is updated
dmitry-lipetsk Mar 1, 2025
d843df6
The old behaviour of RaiseError.UtilityExitedWithNonZeroCode is restored
dmitry-lipetsk Mar 1, 2025
afb4a3d
Merge remote-tracking branch 'origin/D20250301_01--os_ops-exec_comman…
dmitry-lipetsk Mar 1, 2025
25c7c0f
TestRemoteOperations::test_is_executable_true is corrected
dmitry-lipetsk Mar 1, 2025
c4cbaf3
Merge branch 'D20250301_002--remote_ops_tests-test_is_executable_true…
dmitry-lipetsk Mar 1, 2025
491b693
Merge branch 'master' into D20250226_001--ci_and_ssh
dmitry-lipetsk Mar 1, 2025
5cec260
xxx::test_logging is corrected (local, remote)
dmitry-lipetsk Mar 2, 2025
6a70944
Merge remote-tracking branch 'origin/D20250302_001--test_logging' int…
dmitry-lipetsk Mar 2, 2025
d15e29c
TEST_FILTER is added
dmitry-lipetsk Mar 2, 2025
9329f22
CI on Ubuntu 24.04 runs all the tests
dmitry-lipetsk Mar 2, 2025
62ae15c
Merge branch 'master' into D20250226_001--ci_and_ssh
dmitry-lipetsk Mar 2, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 15 additions & 20 deletions testgres/operations/local_ops.py
Original file line number Diff line number Diff line change
Expand Up @@ -23,20 +23,6 @@
from distutils import rmtree

CMD_TIMEOUT_SEC = 60
error_markers = [b'error', b'Permission denied', b'fatal']
err_out_markers = [b'Failure']


def has_errors(output=None, error=None):
if output:
if isinstance(output, str):
output = output.encode(get_default_encoding())
return any(marker in output for marker in err_out_markers)
if error:
if isinstance(error, str):
error = error.encode(get_default_encoding())
return any(marker in error for marker in error_markers)
return False


class LocalOperations(OsOperations):
Expand Down Expand Up @@ -134,19 +120,28 @@ def exec_command(self, cmd, wait_exit=False, verbose=False, expect_error=False,
process, output, error = self._run_command(cmd, shell, input, stdin, stdout, stderr, get_process, timeout, encoding)
if get_process:
return process
if not ignore_errors and ((process.returncode != 0 or has_errors(output=output, error=error)) and not expect_error):

if expect_error:
if process.returncode == 0:
raise InvalidOperationException("We expected an execution error.")
elif ignore_errors:
pass
elif process.returncode == 0:
pass
else:
assert not expect_error
assert not ignore_errors
assert process.returncode != 0
RaiseError.UtilityExitedWithNonZeroCode(
cmd=cmd,
exit_code=process.returncode,
msg_arg=error or output,
error=error,
out=output
)
out=output)

if verbose:
return process.returncode, output, error
else:
return output

return output

# Environment setup
def environ(self, var_name):
Expand Down
47 changes: 12 additions & 35 deletions testgres/operations/raise_error.py
Original file line number Diff line number Diff line change
@@ -1,50 +1,27 @@
from ..exceptions import ExecUtilException
from .helpers import Helpers


class RaiseError:
@staticmethod
def UtilityExitedWithNonZeroCode(cmd, exit_code, msg_arg, error, out):
def UtilityExitedWithNonZeroCode(cmd, exit_code, error, out):
assert type(exit_code) == int # noqa: E721

msg_arg_s = __class__._TranslateDataIntoString(msg_arg).strip()
assert type(msg_arg_s) == str # noqa: E721

if msg_arg_s == "":
msg_arg_s = "#no_error_message"

message = "Utility exited with non-zero code. Error: `" + msg_arg_s + "`"
raise ExecUtilException(
message=message,
message="Utility exited with non-zero code.",
command=cmd,
exit_code=exit_code,
out=out,
error=error)

@staticmethod
def _TranslateDataIntoString(data):
if type(data) == bytes: # noqa: E721
return __class__._TranslateDataIntoString__FromBinary(data)

return str(data)

@staticmethod
def _TranslateDataIntoString__FromBinary(data):
assert type(data) == bytes # noqa: E721

try:
return data.decode(Helpers.GetDefaultEncoding())
except UnicodeDecodeError:
pass

return "#cannot_decode_text"

@staticmethod
def _BinaryIsASCII(data):
assert type(data) == bytes # noqa: E721

for b in data:
if not (b >= 0 and b <= 127):
return False
def CommandExecutionError(cmd, exit_code, message, error, out):
assert type(exit_code) == int # noqa: E721
assert type(message) == str # noqa: E721
assert message != ""

return True
raise ExecUtilException(
message=message,
command=cmd,
exit_code=exit_code,
out=out,
error=error)
102 changes: 67 additions & 35 deletions testgres/operations/remote_ops.py
Original file line number Diff line number Diff line change
Expand Up @@ -100,41 +100,39 @@ def exec_command(self, cmd, wait_exit=False, verbose=False, expect_error=False,
return process

try:
result, error = process.communicate(input=input_prepared, timeout=timeout)
output, error = process.communicate(input=input_prepared, timeout=timeout)
except subprocess.TimeoutExpired:
process.kill()
raise ExecUtilException("Command timed out after {} seconds.".format(timeout))

exit_status = process.returncode

assert type(result) == bytes # noqa: E721
assert type(output) == bytes # noqa: E721
assert type(error) == bytes # noqa: E721

if not error:
error_found = False
else:
error_found = exit_status != 0 or any(
marker in error for marker in [b'error', b'Permission denied', b'fatal', b'No such file or directory']
)

assert type(error_found) == bool # noqa: E721

if encoding:
result = result.decode(encoding)
output = output.decode(encoding)
error = error.decode(encoding)

if not ignore_errors and error_found and not expect_error:
if expect_error:
if process.returncode == 0:
raise InvalidOperationException("We expected an execution error.")
elif ignore_errors:
pass
elif process.returncode == 0:
pass
else:
assert not expect_error
assert not ignore_errors
assert process.returncode != 0
RaiseError.UtilityExitedWithNonZeroCode(
cmd=cmd,
exit_code=exit_status,
msg_arg=error,
exit_code=process.returncode,
error=error,
out=result)
out=output)

if verbose:
return exit_status, result, error
else:
return result
return process.returncode, output, error

return output

# Environment setup
def environ(self, var_name: str) -> str:
Expand Down Expand Up @@ -165,8 +163,30 @@ def find_executable(self, executable):

def is_executable(self, file):
# Check if the file is executable
is_exec = self.exec_command("test -x {} && echo OK".format(file))
return is_exec == b"OK\n"
command = ["test", "-x", file]

exit_status, output, error = self.exec_command(cmd=command, encoding=get_default_encoding(), ignore_errors=True, verbose=True)

assert type(output) == str # noqa: E721
assert type(error) == str # noqa: E721

if exit_status == 0:
return True

if exit_status == 1:
return False

errMsg = "Test operation returns an unknown result code: {0}. File name is [{1}].".format(
exit_status,
file)

RaiseError.UtilityExitedWithNonZeroCode(
cmd=command,
exit_code=exit_status,
msg_arg=errMsg,
error=error,
out=output
)

def set_env(self, var_name: str, var_val: str):
"""
Expand Down Expand Up @@ -251,15 +271,21 @@ def mkdtemp(self, prefix=None):
else:
command = ["mktemp", "-d"]

exit_status, result, error = self.exec_command(command, verbose=True, encoding=get_default_encoding(), ignore_errors=True)
exec_exitcode, exec_output, exec_error = self.exec_command(command, verbose=True, encoding=get_default_encoding(), ignore_errors=True)

assert type(result) == str # noqa: E721
assert type(error) == str # noqa: E721
assert type(exec_exitcode) == int # noqa: E721
assert type(exec_output) == str # noqa: E721
assert type(exec_error) == str # noqa: E721

if exit_status != 0:
raise ExecUtilException("Could not create temporary directory. Error code: {0}. Error message: {1}".format(exit_status, error))
if exec_exitcode != 0:
RaiseError.CommandExecutionError(
cmd=command,
exit_code=exec_exitcode,
message="Could not create temporary directory.",
error=exec_error,
out=exec_output)

temp_dir = result.strip()
temp_dir = exec_output.strip()
return temp_dir

def mkstemp(self, prefix=None):
Expand All @@ -273,15 +299,21 @@ def mkstemp(self, prefix=None):
else:
command = ["mktemp"]

exit_status, result, error = self.exec_command(command, verbose=True, encoding=get_default_encoding(), ignore_errors=True)
exec_exitcode, exec_output, exec_error = self.exec_command(command, verbose=True, encoding=get_default_encoding(), ignore_errors=True)

assert type(result) == str # noqa: E721
assert type(error) == str # noqa: E721
assert type(exec_exitcode) == int # noqa: E721
assert type(exec_output) == str # noqa: E721
assert type(exec_error) == str # noqa: E721

if exit_status != 0:
raise ExecUtilException("Could not create temporary file. Error code: {0}. Error message: {1}".format(exit_status, error))
if exec_exitcode != 0:
RaiseError.CommandExecutionError(
cmd=command,
exit_code=exec_exitcode,
message="Could not create temporary file.",
error=exec_error,
out=exec_output)

temp_file = result.strip()
temp_file = exec_output.strip()
return temp_file

def copytree(self, src, dst):
Expand Down
13 changes: 7 additions & 6 deletions testgres/utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@
from .config import testgres_config as tconf
from .operations.os_ops import OsOperations
from .operations.remote_ops import RemoteOperations
from .operations.helpers import Helpers as OsHelpers

# rows returned by PG_CONFIG
_pg_config_data = {}
Expand Down Expand Up @@ -79,13 +80,13 @@ def execute_utility2(os_ops: OsOperations, args, logfile=None, verbose=False, ig
assert type(verbose) == bool # noqa: E721
assert type(ignore_errors) == bool # noqa: E721

exit_status, out, error = os_ops.exec_command(args, verbose=True, ignore_errors=ignore_errors)
# decode result
exit_status, out, error = os_ops.exec_command(
args,
verbose=True,
ignore_errors=ignore_errors,
encoding=OsHelpers.GetDefaultEncoding())

out = '' if not out else out
if isinstance(out, bytes):
out = out.decode('utf-8')
if isinstance(error, bytes):
error = error.decode('utf-8')

# write new log entry if possible
if logfile:
Expand Down
5 changes: 3 additions & 2 deletions tests/test_local.py
Original file line number Diff line number Diff line change
Expand Up @@ -40,10 +40,11 @@ def test_exec_command_failure(self):
try:
self.operations.exec_command(cmd, wait_exit=True, shell=True)
except ExecUtilException as e:
error = e.message
assert e.message == "Utility exited with non-zero code."
assert type(e.error) == bytes # noqa: E721
assert e.error.strip() == b"/bin/sh: 1: nonexistent_command: not found"
break
raise Exception("We wait an exception!")
assert error == "Utility exited with non-zero code. Error: `/bin/sh: 1: nonexistent_command: not found`"

def test_exec_command_failure__expect_error(self):
"""
Expand Down
10 changes: 6 additions & 4 deletions tests/test_remote.py
Original file line number Diff line number Diff line change
Expand Up @@ -38,10 +38,11 @@ def test_exec_command_failure(self):
try:
self.operations.exec_command(cmd, verbose=True, wait_exit=True)
except ExecUtilException as e:
error = e.message
assert e.message == "Utility exited with non-zero code."
assert type(e.error) == bytes # noqa: E721
assert e.error.strip() == b"bash: line 1: nonexistent_command: command not found"
break
raise Exception("We wait an exception!")
assert error == 'Utility exited with non-zero code. Error: `bash: line 1: nonexistent_command: command not found`'

def test_exec_command_failure__expect_error(self):
"""
Expand Down Expand Up @@ -108,10 +109,11 @@ def test_makedirs_and_rmdirs_failure(self):
try:
self.operations.rmdirs(path, verbose=True)
except ExecUtilException as e:
error = e.message
assert e.message == "Utility exited with non-zero code."
assert type(e.error) == bytes # noqa: E721
assert e.error.strip() == b"rm: cannot remove '/root/test_dir': Permission denied"
break
raise Exception("We wait an exception!")
assert error == "Utility exited with non-zero code. Error: `rm: cannot remove '/root/test_dir': Permission denied`"

def test_listdir(self):
"""
Expand Down
43 changes: 24 additions & 19 deletions tests/test_simple_remote.py
Original file line number Diff line number Diff line change
Expand Up @@ -178,27 +178,32 @@ def test_init__unk_LANG_and_LC_CTYPE(self):
assert os.environ.get("LC_CTYPE") == unkData[1]
assert not ("LC_COLLATE" in os.environ.keys())

while True:
assert os.getenv('LANG') == unkData[0]
assert os.getenv('LANGUAGE') is None
assert os.getenv('LC_CTYPE') == unkData[1]
assert os.getenv('LC_COLLATE') is None

exc: ExecUtilException = None
with __class__.helper__get_node() as node:
try:
with __class__.helper__get_node():
pass
except ExecUtilException as e:
#
# Example of an error message:
#
# warning: setlocale: LC_CTYPE: cannot change locale (UNKNOWN_CTYPE): No such file or directory
# postgres (PostgreSQL) 14.12
#
errMsg = str(e)

logging.info("Error message is: {0}".format(errMsg))

assert "LC_CTYPE" in errMsg
assert unkData[1] in errMsg
assert "warning: setlocale: LC_CTYPE: cannot change locale (" + unkData[1] + "): No such file or directory" in errMsg
assert ("postgres" in errMsg) or ("PostgreSQL" in errMsg)
break
node.init() # IT RAISES!
except InitNodeException as e:
exc = e.__cause__
assert exc is not None
assert isinstance(exc, ExecUtilException)

if exc is None:
raise Exception("We expected an error!")

assert isinstance(exc, ExecUtilException)

errMsg = str(exc)
logging.info("Error message is {0}: {1}".format(type(exc).__name__, errMsg))

assert "warning: setlocale: LC_CTYPE: cannot change locale (" + unkData[1] + ")" in errMsg
assert "initdb: error: invalid locale settings; check LANG and LC_* environment variables" in errMsg
continue

finally:
__class__.helper__restore_envvar("LANG", prev_LANG)
__class__.helper__restore_envvar("LANGUAGE", prev_LANGUAGE)
Expand Down