Skip to content

[Translation] Allow translation:extract to sort messages with --force #52938

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -87,7 +87,7 @@ protected function configure(): void
new InputOption('force', null, InputOption::VALUE_NONE, 'Should the extract be done'),
new InputOption('clean', null, InputOption::VALUE_NONE, 'Should clean not found messages'),
new InputOption('domain', null, InputOption::VALUE_OPTIONAL, 'Specify the domain to extract'),
new InputOption('sort', null, InputOption::VALUE_OPTIONAL, 'Return list of messages sorted alphabetically (only works with --dump-messages)', 'asc'),
new InputOption('sort', null, InputOption::VALUE_OPTIONAL, 'Return list of messages sorted alphabetically'),
Copy link
Contributor Author

@marien-probesys marien-probesys Dec 18, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm a bit unsure about removing the default value. As explained in my commit, it's to allow "null" value in order to keep the behaviour from before when using --force. I made sure to default to "asc" when using --dump-messages, but I may miss some issues.

new InputOption('as-tree', null, InputOption::VALUE_OPTIONAL, 'Dump the messages as a tree-like structure: The given value defines the level where to switch to inline YAML'),
])
->setHelp(<<<'EOF'
Expand Down Expand Up @@ -116,6 +116,7 @@ protected function configure(): void
You can dump a tree-like structure using the yaml format with <comment>--as-tree</> flag:

<info>php %command.full_name% --force --format=yaml --as-tree=3 en AcmeBundle</info>
<info>php %command.full_name% --force --format=yaml --sort=asc --as-tree=3 fr</info>

EOF
)
Expand Down Expand Up @@ -234,19 +235,22 @@ protected function execute(InputInterface $input, OutputInterface $output): int

$domainMessagesCount = \count($list);

if ($sort = $input->getOption('sort')) {
$sort = strtolower($sort);
if (!\in_array($sort, self::SORT_ORDERS, true)) {
$errorIo->error(['Wrong sort order', 'Supported formats are: '.implode(', ', self::SORT_ORDERS).'.']);
$sort = $input->getOption('sort');
if (null === $sort) {
$sort = 'asc';
}

$sort = strtolower($sort);
if (!\in_array($sort, self::SORT_ORDERS, true)) {
$errorIo->error(['Wrong sort order', 'Supported formats are: '.implode(', ', self::SORT_ORDERS).'.']);

return 1;
}
return 1;
}

if (self::DESC === $sort) {
rsort($list);
} else {
sort($list);
}
if (self::DESC === $sort) {
rsort($list);
} else {
sort($list);
}

$io->section(sprintf('Messages extracted for domain "<info>%s</info>" (%d message%s)', $domain, $domainMessagesCount, $domainMessagesCount > 1 ? 's' : ''));
Expand Down Expand Up @@ -277,7 +281,14 @@ protected function execute(InputInterface $input, OutputInterface $output): int
$bundleTransPath = end($transPaths);
}

$this->writer->write($operation->getResult(), $format, ['path' => $bundleTransPath, 'default_locale' => $this->defaultLocale, 'xliff_version' => $xliffVersion, 'as_tree' => $input->getOption('as-tree'), 'inline' => $input->getOption('as-tree') ?? 0]);
$this->writer->write($operation->getResult(), $format, [
'path' => $bundleTransPath,
'default_locale' => $this->defaultLocale,
'xliff_version' => $xliffVersion,
'as_tree' => $input->getOption('as-tree'),
'inline' => $input->getOption('as-tree') ?? 0,
'sort' => $input->getOption('sort'),
]);

if (true === $input->getOption('dump-messages')) {
$resultMessage .= ' and translation files were updated';
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,9 @@ public function formatCatalogue(MessageCatalogue $messages, string $domain, arra
{
$handle = fopen('php://memory', 'r+');

foreach ($messages->all($domain) as $source => $target) {
$sort = $options['sort'] ?? null;

foreach ($messages->all($domain, $sort) as $source => $target) {
fputcsv($handle, [$source, $target], $this->delimiter, $this->enclosure);
}

Expand Down
6 changes: 4 additions & 2 deletions src/Symfony/Component/Translation/Dumper/FileDumper.php
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,8 @@ public function dump(MessageCatalogue $messages, array $options = []): void
throw new InvalidArgumentException('The file dumper needs a path option.');
}

$sort = $options['sort'] ?? null;

// save a file for each domain
foreach ($messages->getDomains() as $domain) {
$fullpath = $options['path'].'/'.$this->getRelativePath($domain, $messages->getLocale());
Expand All @@ -55,7 +57,7 @@ public function dump(MessageCatalogue $messages, array $options = []): void
}

$intlDomain = $domain.MessageCatalogue::INTL_DOMAIN_SUFFIX;
$intlMessages = $messages->all($intlDomain);
$intlMessages = $messages->all($intlDomain, $sort);

if ($intlMessages) {
$intlPath = $options['path'].'/'.$this->getRelativePath($intlDomain, $messages->getLocale());
Expand All @@ -64,7 +66,7 @@ public function dump(MessageCatalogue $messages, array $options = []): void
$messages->replace([], $intlDomain);

try {
if ($messages->all($domain)) {
if ($messages->all($domain, $sort)) {
file_put_contents($fullpath, $this->formatCatalogue($messages, $domain, $options));
}
continue;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -25,8 +25,9 @@ class IcuResFileDumper extends FileDumper
public function formatCatalogue(MessageCatalogue $messages, string $domain, array $options = []): string
{
$data = $indexes = $resources = '';
$sort = $options['sort'] ?? null;

foreach ($messages->all($domain) as $source => $target) {
foreach ($messages->all($domain, $sort) as $source => $target) {
$indexes .= pack('v', \strlen($data) + 28);
$data .= $source."\0";
}
Expand All @@ -35,7 +36,7 @@ public function formatCatalogue(MessageCatalogue $messages, string $domain, arra

$keyTop = $this->getPosition($data);

foreach ($messages->all($domain) as $source => $target) {
foreach ($messages->all($domain, $sort) as $source => $target) {
$resources .= pack('V', $this->getPosition($data));

$data .= pack('V', \strlen($target))
Expand All @@ -46,7 +47,7 @@ public function formatCatalogue(MessageCatalogue $messages, string $domain, arra

$resOffset = $this->getPosition($data);

$data .= pack('v', \count($messages->all($domain)))
$data .= pack('v', \count($messages->all($domain, $sort)))
.$indexes
.$this->writePadding($data)
.$resources
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,9 @@ public function formatCatalogue(MessageCatalogue $messages, string $domain, arra
{
$output = '';

foreach ($messages->all($domain) as $source => $target) {
$sort = $options['sort'] ?? null;

foreach ($messages->all($domain, $sort) as $source => $target) {
$escapeTarget = str_replace('"', '\"', $target);
$output .= $source.'="'.$escapeTarget."\"\n";
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -23,8 +23,9 @@ class JsonFileDumper extends FileDumper
public function formatCatalogue(MessageCatalogue $messages, string $domain, array $options = []): string
{
$flags = $options['json_encoding'] ?? \JSON_PRETTY_PRINT;
$sort = $options['sort'] ?? null;

return json_encode($messages->all($domain), $flags);
return json_encode($messages->all($domain, $sort), $flags);
Copy link
Contributor Author

@marien-probesys marien-probesys Dec 18, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know what to do with the Psalm errors on this line. They are not related to my PR so I would say that I should not change anything here. What do you say?

}

protected function getExtension(): string
Expand Down
4 changes: 3 additions & 1 deletion src/Symfony/Component/Translation/Dumper/MoFileDumper.php
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,9 @@ public function formatCatalogue(MessageCatalogue $messages, string $domain, arra
$offsets = [];
$size = 0;

foreach ($messages->all($domain) as $source => $target) {
$sort = $options['sort'] ?? null;

foreach ($messages->all($domain, $sort) as $source => $target) {
$offsets[] = array_map('strlen', [$sources, $source, $targets, $target]);
$sources .= "\0".$source;
$targets .= "\0".$target;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,9 @@ class PhpFileDumper extends FileDumper
{
public function formatCatalogue(MessageCatalogue $messages, string $domain, array $options = []): string
{
return "<?php\n\nreturn ".var_export($messages->all($domain), true).";\n";
$sort = $options['sort'] ?? null;

return "<?php\n\nreturn ".var_export($messages->all($domain, $sort), true).";\n";
}

protected function getExtension(): string
Expand Down
4 changes: 3 additions & 1 deletion src/Symfony/Component/Translation/Dumper/PoFileDumper.php
Original file line number Diff line number Diff line change
Expand Up @@ -29,8 +29,10 @@ public function formatCatalogue(MessageCatalogue $messages, string $domain, arra
$output .= '"Language: '.$messages->getLocale().'\n"'."\n";
$output .= "\n";

$sort = $options['sort'] ?? null;

$newLine = false;
foreach ($messages->all($domain) as $source => $target) {
foreach ($messages->all($domain, $sort) as $source => $target) {
if ($newLine) {
$output .= "\n";
} else {
Expand Down
4 changes: 3 additions & 1 deletion src/Symfony/Component/Translation/Dumper/QtFileDumper.php
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,9 @@ public function formatCatalogue(MessageCatalogue $messages, string $domain, arra
$context = $ts->appendChild($dom->createElement('context'));
$context->appendChild($dom->createElement('name', $domain));

foreach ($messages->all($domain) as $source => $target) {
$sort = $options['sort'] ?? null;

foreach ($messages->all($domain, $sort) as $source => $target) {
$message = $context->appendChild($dom->createElement('message'));
$metadata = $messages->getMetadata($source, $domain);
if (isset($metadata['sources'])) {
Expand Down
12 changes: 8 additions & 4 deletions src/Symfony/Component/Translation/Dumper/XliffFileDumper.php
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ public function formatCatalogue(MessageCatalogue $messages, string $domain, arra
return $this->dumpXliff1($defaultLocale, $messages, $domain, $options);
}
if ('2.0' === $xliffVersion) {
return $this->dumpXliff2($defaultLocale, $messages, $domain);
return $this->dumpXliff2($defaultLocale, $messages, $domain, $options);
}

throw new InvalidArgumentException(sprintf('No support implemented for dumping XLIFF version "%s".', $xliffVersion));
Expand All @@ -56,6 +56,8 @@ protected function getExtension(): string

private function dumpXliff1(string $defaultLocale, MessageCatalogue $messages, ?string $domain, array $options = []): string
{
$sort = $options['sort'] ?? null;

$toolInfo = ['tool-id' => 'symfony', 'tool-name' => 'Symfony'];
if (\array_key_exists('tool_info', $options)) {
$toolInfo = array_merge($toolInfo, $options['tool_info']);
Expand Down Expand Up @@ -90,7 +92,7 @@ private function dumpXliff1(string $defaultLocale, MessageCatalogue $messages, ?
}

$xliffBody = $xliffFile->appendChild($dom->createElement('body'));
foreach ($messages->all($domain) as $source => $target) {
foreach ($messages->all($domain, $sort) as $source => $target) {
$translation = $dom->createElement('trans-unit');

$translation->setAttribute('id', strtr(substr(base64_encode(hash('sha256', $source, true)), 0, 7), '/+', '._'));
Expand Down Expand Up @@ -137,8 +139,10 @@ private function dumpXliff1(string $defaultLocale, MessageCatalogue $messages, ?
return $dom->saveXML();
}

private function dumpXliff2(string $defaultLocale, MessageCatalogue $messages, ?string $domain): string
private function dumpXliff2(string $defaultLocale, MessageCatalogue $messages, ?string $domain, array $options = []): string
{
$sort = $options['sort'] ?? null;

$dom = new \DOMDocument('1.0', 'utf-8');
$dom->formatOutput = true;

Expand All @@ -165,7 +169,7 @@ private function dumpXliff2(string $defaultLocale, MessageCatalogue $messages, ?
}
}

foreach ($messages->all($domain) as $source => $target) {
foreach ($messages->all($domain, $sort) as $source => $target) {
$translation = $dom->createElement('unit');
$translation->setAttribute('id', strtr(substr(base64_encode(hash('sha256', $source, true)), 0, 7), '/+', '._'));

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,9 @@ public function formatCatalogue(MessageCatalogue $messages, string $domain, arra
throw new LogicException('Dumping translations in the YAML format requires the Symfony Yaml component.');
}

$data = $messages->all($domain);
$sort = $options['sort'] ?? null;

$data = $messages->all($domain, $sort);

if (isset($options['as_tree']) && $options['as_tree']) {
$data = ArrayConverter::expandToTree($data);
Expand Down
28 changes: 23 additions & 5 deletions src/Symfony/Component/Translation/MessageCatalogue.php
Original file line number Diff line number Diff line change
Expand Up @@ -55,26 +55,44 @@ public function getDomains(): array
return array_values($domains);
}

public function all(string $domain = null): array
public function all(string $domain = null, ?string $sort = null): array
{
$sort = $sort ? strtolower($sort) : null;

if (null !== $domain) {
// skip messages merge if intl-icu requested explicitly
if (str_ends_with($domain, self::INTL_DOMAIN_SUFFIX)) {
return $this->messages[$domain] ?? [];
$messages = $this->messages[$domain] ?? [];
} else {
$messages = ($this->messages[$domain.self::INTL_DOMAIN_SUFFIX] ?? []) + ($this->messages[$domain] ?? []);
}

if ('desc' === $sort) {
krsort($messages);
} elseif ('asc' === $sort) {
ksort($messages);
}

return ($this->messages[$domain.self::INTL_DOMAIN_SUFFIX] ?? []) + ($this->messages[$domain] ?? []);
return $messages;
}

$allMessages = [];

foreach ($this->messages as $domain => $messages) {
if (str_ends_with($domain, self::INTL_DOMAIN_SUFFIX)) {
$domain = substr($domain, 0, -\strlen(self::INTL_DOMAIN_SUFFIX));
$allMessages[$domain] = $messages + ($allMessages[$domain] ?? []);
$messages = $messages + ($allMessages[$domain] ?? []);
} else {
$allMessages[$domain] = ($allMessages[$domain] ?? []) + $messages;
$messages = ($allMessages[$domain] ?? []) + $messages;
}

if ('desc' === $sort) {
krsort($messages);
} elseif ('asc' === $sort) {
ksort($messages);
}
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should the method throw an error if the value of sort is invalid? I like not being too strict and doing nothing when the value is different, but it's less clear if a typo is made (e.g. passing the value acs instead of asc)


$allMessages[$domain] = $messages;
}

return $allMessages;
Expand Down