Bug 1084015

Created attachment 762752 [details]
problems in en_GB

Created attachment 762753 [details]
tool that made the extraction

Tracking in YaST scrum board.

Created attachment 799989 [details]
recent en_US hotfixes to merge in master

I just did a review and deleted all en_US po files with no translations:

git rm $(for i in */en_US.po ; do echo -n $i:\ ; LANG=C msgfmt --statistics $i -o /dev/null ; done 2>&1 | grep " 0 translated messages" | sed 's/:.*//')

It decreased number of affected strings.

The simple script is not capable to detect msgid == msgstr.

So I wrote a simple msgfilter script:
#!/bin/bash
MSGFILTER_MSGSTR="$(cat)"
if test "$MSGFILTER_MSGID" != "$MSGFILTER_MSGSTR" ; then
	echo -n "$MSGFILTER_MSGSTR"
fi

Playing with msgfilter and msgattrib --no-fuzzy --no-obsolete --translated, I got a final list of strings that need fix.

Here is the final file.

Statistics of the work that needs to be merged to the source code:
apparmor: 2 messages.
autoinst: 2 messages.
base: 8 messages.
country: 2 messages.
dhcp-server: 3 messages.
firewall-services: 1 message.
firstboot: 1 message.
http-server: 3 messages.
installation: 2 messages.
kdump: 3 messages.
ldap: 12 messages.
nis_server: 1 message.
qt-pkg: 1 message.
samba-server: 2 messages.
services-manager: 1 message.
sudo: 25 messages.
support: 17 messages.
update: 1 message.

Still an issue.

Please finish the work and integrate these strings to sources. With exception of yast2-sudo, it is a few minutes of work.

for i in */en_US.po ; do
  msgfilter -i $i -o $i.new ./filter.sh
  rm $i
  msgattrib --no-fuzzy --no-obsolete --translated $i.new -o $i
  echo -n "$i: "
  msgfmt --statistics -o /dev/null $i
done
rm */en_US.po.new

apparmor/en_US.po: 2 translated messages.
autoinst/en_US.po: 2 translated messages.
base/en_US.po: 5 translated messages.
dhcp-server/en_US.po: 3 translated messages.
firewall-services/en_US.po: 1 translated message.
http-server/en_US.po: 3 translated messages.
installation/en_US.po: 2 translated messages.
kdump/en_US.po: 3 translated messages.
qt-pkg/en_US.po: 1 translated message.
samba-server/en_US.po: 2 translated messages.
sudo/en_US.po: 25 translated messages.
update/en_US.po: 1 translated message.

Created attachment 827553 [details]
po files with en_US changes only

This po file contains en_US hotfixes in the recent master.

Last po file edit: 2008-05-16 13:44+0200

Frederic Crozat: It either makes sense to pay for the British English translation, or write a sort of machine "translator" from American English to British English than keeping files intact for years.

See comment 0 point 5 and comment 1 for the list of "translated" string. All other appear in the American English.

What is your opinion?

decision is made by PM, not Release Managers.

We can keep one "English" translation, which is the American flavor.
No need to translate this into British English.

Adding people that touched British English translation in last years:

Some of YaST modules were not updated for 19 years for British English, and none was updated in last 2 years!

As SUSE has no budget for maintenance of British English dialect, keeping orphan en_GB translations makes things worse: inconsistent, and it even introduces inconsistent terminology (AppArmor × AppArmour).

If there will be any volunteer who can maintain British English and/or write automatic generator (it could be reasonably easy with msgfilter), en_GB translations will be kept.

If there is no volunteer for this task, British English translations will be dropped from YaST in a near future (and possibly other SUSE projects), and Brits will consistently see American English instead of experience of mixed American and British English.


Updated version of filter.sh that works around dropping of trailing EOL in strings. It works perfectly for en_GB:
#!/bin/bash
MSGFILTER_MSGSTR="$(cat ; echo -n x)"
MSGFILTER_MSGSTR="${MSGFILTER_MSGSTR%x}"
if test "$MSGFILTER_MSGID" != "$MSGFILTER_MSGSTR" ; then
    echo -n "$MSGFILTER_MSGSTR"
fi

One hint about formatting fixes and changes in the source string: All these fixes should be marked with "fuzzy" to edit for our Translators if the workflow is working correctly. Most Translators are volunteers. We have been improving the format and changed words continuously. German translations are done really fast. So it can be tested in Tumbleweed very early because of so many Translators. Other languages are waiting for the "Go" of the next Leap release because of missing time over the year. Then it depends on the count of Translators, whether they translate only "most important files" or more.
Your listed yast modules are not part of "most important files". But I can not believe that all Translations should be wrong...

I have found some places with incorrect format in translations. I will write a call for reviewing all yast translations.

@Stanislav: I have a question about listed wrong syntax. We had a feature in Weblate in the past: If we would not have the same syntax, there was a syntax error listed. I have found wrong html syntax and that was no error message. Is that disabled?

Sarah Kriesch: This is probably a misunderstanding.

This bug discusses exclusively the wrong English, not the translation to other languages.

You have been added to the Cc, as you are one of people mentioned as British English (en_GB) translator. British English translation is so bad, that it makes sense to completely remove it or disable, or at least disable it for some modules.

Other translations to major languages are in a very good shape.


And yes, when the en_US translation will be incorporated to the source code, all translations will be marked as fuzzy. Translators will have to check, whether the English change has to be reflected. (There is another technical possibility: If the developer is sure that the English change is just a typo fix, then it is possible to fix the source string in all po files. But I am afraid that it is over-complicated and developers will not do it.)


Regarding the format problems: This is a technical issue. YaST migrated to Ruby. But ruby-gettext does not support format check yet, and nor weblate nor gettext support ruby-format tag. So Weblate cannot verify string format online. But we have a temporary solution for it: https://github.com/yast/yast-translations/blob/master/tools/check/check_po_files.rb It allows offline checks, and it disables broken strings.

Extended rationale, why NEVER USE en_US "translation" for hotfixes:


Please don't do it! It does not work as expected:

1) It is USA-only fix. English users outside USA (e. g. en_GB and 17 other English locales) see the broken string.

2) Non-English users see broken string as well. Translators translate the source string, not the English hotfix. It easily results in a bad formatting in all locales except en_US (but including C).

Example: yast apparmor src/include/apparmor/profile_dialogs.rb:1487
Source string produces confusing "?" line:
---
 ?
After this operation the AppArmor module will reload the profile set.
---

en_US hotfix from year 2002 removes it:
---
After this operation the AppArmor module will reload the profile set.
---

But we have a broken translation of this string to a more than 40 languages.

en_US hotfix in 2002:
#: src/include/apparmor/profile_dialogs.rb:1487
msgid ""
" ?\n"
"After this operation the AppArmor module will reload the profile set."
msgstr "After this operation the AppArmor module will reload the profile set."

Czech translation in 2020:
#: src/include/apparmor/profile_dialogs.rb:1487
msgid ""
" ?\n"
"After this operation the AppArmor module will reload the profile set."
msgstr ""
" ?\n"
"Po dokončení této operace načte AppArmor modul znovu sadu profilů."

Result in cs_CZ (and >40 other locales):
---
 ?
Po dokončení této operace načte AppArmor modul znovu sadu profilů.
---

In general, you are right, Standa!  Nevertheless en_US "translation" for hotfixes are fine for cosmetic tweaks.  If in doubt whether it's purely cosmetic or not, do not provide an en_US "translation".  And then you must take the time to fix the source code for the next release.

But for a broken string, never ever provide an en_US "translation".  Stanislav explained the reason why.

False alarm for textdomain qt-pkg:


#. Translators: This is a combo box where the user can apply a secondary filter
#. in addition to the primary filter by repository - one of
#. "All packages", "RPM groups", "search", "summary"
#.
#. And yes, the colon really belongs there since this is one of the very
#. few cases where a combo box label is left to the combo box rather than
#. above it.
#: src/YQPkgSecondaryFilterView.cc:103
msgid  "&Secondary Filter:"
msgstr "&Secondary Filter"


This translation is plain wrong; the comment above explicitly says so. I had written it back many years ago in anticipation of just this kind of thing.

The colon is intentional and required. Removing it introduces a bug.
Please remove this translation (it's the only one for this textdomain qt-pkg).

False alarm for textdomain samba-server:

#. translators: progress stage
#: src/modules/SambaServer.pm:155
msgid  "Read Samba service role settings"
msgstr "Read samba service role settings"

#. translators: progress stage
#: src/modules/SambaServer.pm:170
msgid  "Reading Samba service role settings..."
msgstr "Reading samba service role settings..."


Wrong translation: "Samba" is a name here. It's not the dance, it's the software called "Samba", so it needs to be capitalized. It is also capitalized everywhere else in all user-visible texts, so this change even introduces inconsistencies.

Also compare with

  https://en.wikipedia.org/wiki/Samba_(software)

as well as a ton of other web articles about it.
It is capitalized. Always.

Textdomain "apparmor":

PR: https://github.com/yast/yast-apparmor/pull/49

The original authors had pieced together a longer message with variables (ouch!) from smaller pieces, thus the weird dangling question mark:


#: src/include/apparmor/profile_dialogs.rb:1484
msgid "Are you sure you want to delete the profile "
msgstr "Are you sure you want to delete the profile? "

#: src/include/apparmor/profile_dialogs.rb:1487
msgid ""
" ?\n"
"After this operation the AppArmor module will reload the profile set."
msgstr "After this operation the AppArmor module will reload the profile set."


Now it properly uses the sformat() function, but that also means that translations should be adapted.

This may be possible to do even when not speaking the language (at least for most Western languages; not so sure about Chinese, Japanese, Korean): Just add the \"%1\"  at the end of the message.

Textdomain "autoinst":

https://github.com/yast/yast-autoinstallation/pull/563

Textdomain "base":

https://github.com/yast/yast-yast2/pull/1023

Textdomain "country":

The only change was newlines at the end of HTML texts after the closing </p> tag. 
That newline is ignored. No change.

Textdomain "dhcp-server":

https://github.com/yast/yast-dhcp-server/pull/49

Textdomain "firstboot":

Again only a newline after a closing </p> tag in HTML text. Such whitespace is ignored in HTML text anyway. No change.

Textdomain "http-server":

https://github.com/yast/yast-http-server/pull/50

Textdomain "installation":

https://github.com/yast/yast-installation/pull/836

Textdomain "firewall-services":

This is now obsolete (starting with SLE-15).
This was for SuSEfirewall2 which is now replaced by firewalld.

Textdomain "kdump":

https://github.com/yast/yast-kdump/pull/113

Since the same class of error was repeated over and over again (missing a the article), I went over all of those texts and found some more and fixed them.

I also fixed some other broken English, so my fixes went beyond what was suggested in this en_US.po file.

Textdomain "ldap":

Whitespace-only changes (trailing newlines) that are for one part not at all visible because it's in HTML text where whitespace is normalized to one space character and in other cases one extra newline in a pop-up dialog.

No changes.

Textdomain "sudo":

https://github.com/yast/yast-sudo/pull/24

Those changes are all about help texts. The English was really badly broken in a lot of places, and weird control characters were strewn in all over the place. Since it's all HTML text, there is zero reason to add '\t' or '\n' for tabs or newlines; that whitespace is normalized to a single blank anyway.

There were no semantic changes, so it should be safe to simply remove any fuzzy markers in all the translations.

Stefan Hundhammer: Thanks. Feel free to delete invalid "translations", either in https://l10n.opensuse.org/languages/en_US/ or in the yast-translations repository.

I will do that for reported bugs.

When pull requests will be accepted, I will re-run the "translation" extraction script.

I will manually remove all false alarms, where en_US "translation" made things worse.

We should end with an empty en_US "translation".

If it was a pure typo, I can adjust the msgid in translations and save translators' work.


Regarding en_GB translation: It looks possible to integrate automatic translator from American English to British English into Weblate. The whole British English YaST translation contains only two "real" translations, the rest are only (partially buggy) sed replacements.


Karl Eichwalder, comment 17: Even for cosmetic tweaks, it is not OK. It works only for en_US locale, and not for en_GB.

yast-translations updated:

Comment 18, qt-pkg:
grep -A1 "Secondary Filter" * | grep msgstr | grep -v '""' | grep -v ':"'
The same problem affects Tamil, the rest are false matches.
Dropped en_US.po.
4b4b9f701f


Comment 19, samba-server:
The review was done by a non-IT people.
Dropping en_US.po
14818ec4f1


Comment 20, apparmor:
I applied a filter to the old repository:
#!/bin/bash
mkdir -p NEW
for FILENAME in *.po ; do
	export FILENAME
	msgmerge $FILENAME apparmor_old.pot -o - |
		msgattrib --no-obsolete -i - -o - |
		msgfilter -i - -o /dev/null ./FILTER0.sh
	msgmerge $FILENAME apparmor_new_cut.pot -o - |
		msgattrib --no-obsolete -i - -o - |
		sed 's/^msgid.*"Are you sure you want to delete the profile "/msgid "Are you sure you want to delete the profile \\"%1\\"?"/' |
		msgfilter -i - -o - ./FILTER1.sh |
		sed '/" ?\\n"/d' >NEW/$FILENAME
	if test $(wc -l $FILENAME.join | sed 's/ .*//') = 0 ; then
		sed -i '/msgid/i#, fuzzy' NEW/$FILENAME
	fi
done
FILTER0.sh:
#!/bin/sh
if echo "$MSGFILTER_MSGID" | fgrep -q "After this operation the AppArmor module will reload the profile set." ; then
	cat >$FILENAME.join
	cat $FILENAME.join
else
	cat
fi
FILTER1.sh:
#!/bin/bash
if echo "$MSGFILTER_MSGID" | fgrep -q "Are you sure you want to delete the profile" ; then
	STRING="$(head -n1 $FILENAME.join)"
	STRING=${STRING# }
	cat
	echo -n "\"%1\"$STRING"
else
	if echo "$MSGFILTER_MSGID" | fgrep -q "After this operation the AppArmor module will reload the profile set" ; then
		cat >/dev/null
		tail -n1 $FILENAME.join
	else
		cat
	fi
fi
and then moved NEW to the new and did:
mkdir -p FINAL
or FILENAME in *.po ; do msgcat --use-first $FILENAME NEW/$FILENAME -o - | msgmerge --previous - apparmor.pot -o FINAL/$FILENAME ; done

e3c16ee96c 5152d434ff


Comment 21, autoinst:
Easy fixes, but we should probably let translators to check translated terminology as well.
5152d434ff


Comment 22, base:

I applied a filter to the old repository:
#!/bin/bash
mkdir -p NEW FILTERED
for FILENAME in *.po ; do
	msgmerge $FILENAME base_cut_old.pot -o - |
		msgattrib --no-obsolete -i - -o - |
		sed '
s/"Continue or cancel the operation?\n"/"Continue or cancel the operation?"/
s/"Create it now?\n"/"Create it now?"/
' |
		msgfilter -i - -o NEW/$FILENAME ./FILTER.sh
sed '
s/contains list of the p/contains a list of p/
s/Enter passphrase to unlock/Enter the passphrase to unlock/
' <$FILENAME | msgmerge --previous - base_full_new.pot -o - | msgattrib --no-fuzzy --no-obsolete -i - -o FILTERED/$FILENAME
done
FILTER.sh:
#!/bin/sh
if echo "$MSGFILTER_MSGID" | grep -q "\(Continue or cancel the operation\|Create it now\)" ; then
	STRING="$(cat)"
	STRING=${STRING%\n}
	echo -n "$STRING"
else
	cat
fi

and then moved NEW to the new and did:
mkdir -p FINAL
for FILENAME in *.po ; do msgcat --use-first $FILENAME FILTERED/$FILENAME NEW/$FILENAME -o - | msgmerge --previous - base.pot -o FINAL/$FILENAME ; done

734d221b51 c8dbf75090


Comment 23, country:
Already not present in the problems file dated 2020-01-15.
Dropping en_US.po
c8dbf75090


Comment 24, dhcp-server:
In the old:
s/that you want DHCP Server/that you want the DHCP Server/
s/consult dhcpd manual page/consult the dhcpd manual page/
The rest needs to be verified by translators.
In the new:
for FILENAME in *.po ; do msgcat --use-first $FILENAME OLD/$FILENAME -o - | msgmerge --previous - dhcp-server.pot -o FINAL/$FILENAME ; done
a949b7f04d


Comment 25, firstboot:
Already not present in the problems file dated 2020-01-15.
Dropping en_US.po
bc3116beb6


Comment 26, http-server:
Will apply
s/When no Server name is defined, hostname will/When no server name is defined, the hostname/
The rest needs to be verified by translators.
TODO (waiting for pull request)


Comment 27, installation:
Applying
s/click on the/click the/g
=> 100% again

By the way, installation also has hard-split lines in many strings:
example:
Instead AutoYaST will dump an\nimage on the harddisk

If you will fix it, a sed script similar to sudo can be applied.

TODO (waiting for pull request)


Comment 28, firewall-services: Please note that firewall-services.pot comes from yast2/library/network/src/yast2-services-translations.rb that is NOT DROPPED!

So maybe YaST maintains dead part of the code that generates this domain. (The same as rpm-groups.)


Comment 29, kdump: Let translators check and fix translations.


Comment 30, ldap: Did you check the latest tarball here (dated 2020-01-15). ldap is not there.

Actually, there is one outdated hotfix that became fuzzy and disappeared:
#. error popup
#: src/LdapPopup.rb:439
#, fuzzy
#| msgid ""
#| "The entered value already exists.\n"
#| "Select another one.\n"
msgid ""
"The value '%1' already exists.\n"
"Please select another one."
msgstr ""
"The entered value already exists.\n"
"Select another one."


Comment 31, sudo: As there are many minor changes, machine fixes of translations are not real.

But I wrote a script that will make translators' work easier. It will drop EOLs and TABs automatically. (Number of new fuzzy strings remains the same, but reject size will be smaller.)

/\\n"$/{
	$!{
		N
		s/\\n"\n"\\t/ "\n"/
		t 1
		P
		D
		:1
	}
}
(It needs to apply multiply times.)
TODO (waiting for pull request)

(In reply to Stanislav Brabec from comment #33)
> Comment 31, sudo: As there are many minor changes, machine fixes of
> translations are not real.
> 
> But I wrote a script that will make translators' work easier. It will drop
> EOLs and TABs automatically. (Number of new fuzzy strings remains the same,
> but reject size will be smaller.)
> 
> /\\n"$/{
> 	$!{
> 		N
> 		s/\\n"\n"\\t/ "\n"/
> 		t 1
> 		P
> 		D
> 		:1
> 	}
> }
> (It needs to apply multiply times.)
> TODO (waiting for pull request)


Careful: This only applies to HTML-like text. In other places newlines are really essential. You can tell by the presence or absence of HTML tags like <p> in the msgid.

Textdomain "update":

https://github.com/yast/yast-update/pull/149

Textdomains "nis-server", "services-manager", "support": 

All just trailing '\n' newlines in HTML texts.
No change.

.

Comment 34, sudo: I agree. After many attempts, I gave it up. The translation would be very complicated. Translators have to do it manually. Over 30 fuzzy strings with lots of changes.
I did only plain msgmerge. 4ff23a85ee

Comment 27, installation:
Applied
s/click on the/click the/g
=> 100% again
c94c6c2bef c9dad12743

Comment 26, http-server:
Applied
s/When no Server name is defined, hostname will/When no server name is defined, the hostname will/
The rest needs to be verified by translators.
9bdaaff929

Comment 36, nis-server, services-manager, support:
c9dad12743

Comment 35, update:
The text changes meaning => translators need to fix the change.
Just drop en_US.po
0533a023aa

kdump
Drop en_US.po
36a6ac2347

ldap
Drop en_US.po. See comment 33 for note.
36a6ac2347

To resolve (from dropped en_US.po):
Drop en_US.po. See next comment.
firewall-services
36a6ac2347

Now yast-translations has no en_US.po. Thanks for you co operation.


There are remaining issues to evaluate:


yast2:

Please evaluate eventual obsolescence of the code in yast2/library/network/src/yast2-services-translations.rb, and eventually evaluate these strings:

#. TRANSLATORS: Description of a Service (File name: vnc-httpd, RPM: tightvnc), used as a common label or an item in table
#: library/network/src/yast2-services-translations.rb:176
msgid "Opens the VNC HTTP ports so that browsers can connect."
msgstr "Opens the VNC HTTP ports so browsers can connect."

. TRANSLATORS: Description of a Service (File name: vnc-server, RPM: tightvnc), used as a common label or an item in table
#: library/network/src/yast2-services-translations.rb:182
#, fuzzy
#| msgid "Opens the VNC HTTP ports so that browsers can connect."
msgid "Open VNC server ports so that viewers can connect."
msgstr "Opens the VNC HTTP ports so browsers can connect."


ldap:

This hotfix is obsolete itself, as the string changed. There was a \n hotfix in past:

#. error popup
#: src/LdapPopup.rb:439
msgid ""
"The entered value already exists.\n"
"Select another one.\n"
msgstr ""
"The entered value already exists.\n"
"Select another one."

(In reply to Stefan Hundhammer from comment #31)
> Textdomain "sudo":

Thanks for working on all these issues.

> There were no semantic changes, so it should be safe to simply remove any
> fuzzy markers in all the translations.

I'd recommend letting translators remove the fuzzy markers even if you fixed just a something simple.

(In reply to Karl Eichwalder from comment #40)
> (In reply to Stefan Hundhammer from comment #31)
> > There were no semantic changes, so it should be safe to simply remove any
> > fuzzy markers in all the translations.
> 
> I'd recommend letting translators remove the fuzzy markers even if you fixed
> just a something simple.

Approved!
It is not wrong to receive some reviews by native speakers (especially a short time before the next Leap release).
Therefore, let us show through that all. 4 eyes can see more than 2. :)

Sarah Kriesch: Karl Eichwalder: I already did machine edits for apparent typo fixes and things that could be done by machine.

Please keep in mind that a simple typo fix would would make fuzzy strings in ~75 languages. It would require 75 translators to inspect each of these fixes. Only 8 of them is paid by SUSE.

I did my best to get translations machine re-enabled. Only for sude, I gave it up, as there is a large number of complex fixes.

You can inspect upper mentioned git ids.

Only this replacement was complicated and potentially fragile:
https://github.com/yast/yast-translations/commit/734d221b51

This is an autogenerated message for OBS integration:
This bug (1084015) was mentioned in
https://build.opensuse.org/request/show/784100 Factory / yast2-trans