Skip to content

RemoteOperations::exec_command explicitly transfers LANG, LANGUAGE and LC_* envvars to the server side #187

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Prev Previous commit
Next Next commit
RemoteOperations::exec_command explicitly transfers LANG, LANGUAGE an…
…d LC_* envvars to the server side

It should help resolve a problem with replacing a LANG variable by ssh-server.

History.

On our internal tests we got a problem on the Debian 11 and PostgresPro STD-13.

One test returned the error from initdb:

initdb: error: collations with different collate and ctype values ("en_US.UTF-8" and "C.UTF-8" accordingly) are not supported by ICU

- TestRunner set variable LANG="C"
- Python set variable LC_CTYPE="C.UTF-8"
- Test call inidb through command "ssh test@localhost inidb -D ...."
- SSH-server replaces LANG with value "en_US.UTF-8" (from etc/default/locale)
- initdb calculate collate through this value of LANG variable and get en_US.UTF-8

So we have that:
- ctype is C.UTF-8
- collate is en_US.UTF-8

ICU on the Debuan-11 (uconv v2.1  ICU 67.1) does not suppot this combination and inidb rturns the error.

This patch generates a new command line for ssh:

ssh test@localhost "LANG=\"...\";LC_xxx=\"...\";<command>"

It resolves this problem with initdb and should help resolve other problems with execution of command through SSH.

Amen.
  • Loading branch information
dmitry-lipetsk committed Feb 20, 2025
commit 2b1db89c0375c1ec3a130e31fa46ab8b7339141e
43 changes: 42 additions & 1 deletion testgres/operations/remote_ops.py
Original file line number Diff line number Diff line change
Expand Up @@ -87,7 +87,12 @@ def exec_command(self, cmd, wait_exit=False, verbose=False, expect_error=False,

assert type(cmd_s) == str # noqa: E721

ssh_cmd = ['ssh', self.ssh_dest] + self.ssh_args + [cmd_s]
cmd_items = __class__._make_exec_env_list()
cmd_items.append(cmd_s)

env_cmd_s = ';'.join(cmd_items)

ssh_cmd = ['ssh', self.ssh_dest] + self.ssh_args + [env_cmd_s]

process = subprocess.Popen(ssh_cmd, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
assert not (process is None)
Expand Down Expand Up @@ -510,6 +515,42 @@ def db_connect(self, dbname, user, password=None, host="localhost", port=5432):
)
return conn

def _make_exec_env_list() -> list[str]:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

тут нужно self, либо добавить @staticmethod

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

self тут не нужен. Это статический метод.

Хорошо, я добавлю этот декоратор.

Я там в других местах тоже статические методы определял...

Тогда я пока только в новом коде этого PR его добавлю, а потом, отдельным коммитом, добавим его во все остальные случаи.

PS. Я этот питон на ходу изучаю, поэтому многие вещи использую неправильно ))))

result = list[str]()
for envvar in os.environ.items():
if not __class__._does_put_envvar_into_exec_cmd(envvar[0]):
continue
qvalue = __class__._quote_envvar(envvar[1])
assert type(qvalue) == str # noqa: E721
result.append(envvar[0] + "=" + qvalue)
continue

return result

sm_envs_for_exec_cmd = ["LANG", "LANGUAGE"]

def _does_put_envvar_into_exec_cmd(name: str) -> bool:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

тут нужно self, либо добавить @staticmethod

assert type(name) == str # noqa: E721
name = name.upper()
if name.startswith("LC_"):
return True
if name in __class__.sm_envs_for_exec_cmd:
return True
return False

def _quote_envvar(value) -> str:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

тут нужно self, либо добавить @staticmethod

assert type(value) == str # noqa: E721
result = "\""
for ch in value:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Тут лучше append использовать https://www.geeksforgeeks.org/difference-between-and-append-in-python/

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Виктория, что-то я не догоняю. Куда тут append использовать?

result - это str. У него нет метода append, но есть поддержка оператора +=, который по смыслу тот же append. Этот += тут и используется.

Copy link
Contributor

@demonolock demonolock Feb 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

типо так


       result = ['"']
        for ch in value:
            if ch == '"':
                result.append('\\"')
            elif ch == '\\':
                result.append('\\\\')
            else:
                result.append(ch)
        result.append('"')
        return ''.join(result)

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ага. Не. Не надо нам такой "оптимизации" )

Во первых у нас тут строка. И её элементы могут хранится непрерывно, ну или по крайней мере питон там может как-то оптимизировать это дело. А если будет массив строк - то однозначно каждый элемент массива будет хранится отдельно. Плюс место под сам массив.

А во вторых у нас тут нет оператора сложения. Используется оператор +=.

С append можно заморочится выше - там я три массива складываю через +.

Но честно - если бы эти бестолочи (я про авторов питона) позволяли писать append(list1).append(list2), я бы его заюзал. А так - в топку.


25 лет назад на .NET считалось дурным тоном складывать строки. Юзайте, говорят, StringBuilder. Но потом все дружно на него забили ))

if ch == "\"":
result += "\\\""
elif ch == "\\":
result += "\\\\"
else:
result += ch
result += "\""
return result


def normalize_error(error):
if isinstance(error, bytes):
Expand Down