Bug 7939: Separate po files for different MARC dialects

This patch implements separate PO files for
different MARC dialects.

It depends on correct filenames, i.e. it will build PO
files using files with/without "unimarc|normarc|marc21"
on their names.

Changes:
A) LangInstaller.pm
Added definitions to create or update xx-YY-{MARCFLAVOR}.po,
minor change to create and install procedure, and modification
of install procedure to handle multiple target dirs.
Updated documentation.

B) Standarization of filenames
STAFF po file is now xx-YY-staff-prog.po
MARC dialects po files are xx-YY-marc-{MARCFLAVOUR}.po

To test:
1) Update po files for your preferred language, ej. nn-NO
cd misc/translator
perl translate update nn-NO

2) Do some copying/renaming
cp po/nn-NO-i-staff-t-prog-v-3006000.po po/nn-NO-marc-UNIMARC.po
cp po/nn-NO-i-staff-t-prog-v-3006000.po po/nn-NO-marc-NORMARC.po
cp po/nn-NO-i-staff-t-prog-v-3006000.po po/nn-NO-marc-MARC21.po
mv po/nn-NO-i-staff-t-prog-v-3006000.po po/nn-NO-staff-prog.po

(most MARC dialect strings are on staff, so we use that as basis)

3) Apply the patch

4) Update again to fix translation files, verbose
perl translate update nn-NO -v

5) Install language, verbose, verify translations
perl translate install nn-NO -v

6) Create translation files
rm po/nn-NO*
perl translate create nn-NO

we must have this list:
po/nn-NO-marc-MARC21.po
po/nn-NO-marc-NORMARC.po
po/nn-NO-marc-UNIMARC.po
po/nn-NO-opac-bootstrap.po
po/nn-NO-pref.po
po/nn-NO-staff-help.po
po/nn-NO-staff-prog.po

Additional tests:
7) Number of msgids
7.a) Before patch and after upgrade, extract and count msgids
for i in $(ls po/nn-NO-*po); \
do msginit -i $i -o nn-old.po --no-translator --no-wrap --locale=nn_NO; \
egrep ^msgid nn-old.po >> old; \
done
sort old | uniq | tee s-old | wc -l > n-old
s-old: have all msgids
n-old: number of msgids

7.b) After patch and after creation of new files
Repeat procedure, diferent files (s-new, n-new)

7.c) Compare (diff s-old snew), they are the same
(save for a strange UNIMARC char in my case, but
it's present on corresponding PO file)

8) Installed dirs/files
8.a) List of EN dirs/files
cd koha-tmpl
find | egrep "/en/" > en

8.b) List of nn-NO dirs/files. After patch and language install
cd koha-tmpl
find | egrep "/nn-NO/" | sed 's|/nn-NO/|/en/|' > nn

8.c) Compare (diff en nn), they are the same

Signed-off-by: Magnus Enger <magnus@enger.priv.no>
Followed the steps outlined by Bernardo, and everything works as
expected. I think the most important points are that "perl translate
create nn-NO" produces the right files, and translating anything in
them, then doing "translate install" makes the translations show
up in the interface. The numbers msgids in the nn-NO correspond
well wit the number of msgids in other sets of .po files.

I bet y'all will be happy when you don't have to see the stupid
Norwegian strings when you translate! ;-)

Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>

diff --git a/misc/translator/LangInstaller.pm b/misc/translator/LangInstaller.pm
index 9e40ede..ad814dd 100644 (file)
--- a/misc/translator/LangInstaller.pm
+++ b/misc/translator/LangInstaller.pm
@@ -95,7 +95,7 @@ sub new {
         {
             name   => 'Intranet prog UI',
             dir    => $context->config('intrahtdocs') . '/prog',
-            suffix => '-i-staff-t-prog-v-3006000.po',
+            suffix => '-staff-prog.po',
         },
         {
             name   => 'Intranet prog help',
@@ -114,6 +114,21 @@ sub new {
         };
     }
 
+    # MARC flavours (hardcoded list)
+    for ( "MARC21", "UNIMARC", "NORMARC" ) {
+        # search for strings on staff & opac marc files
+        my $dirs = $context->config('intrahtdocs') . '/prog';
+        opendir $fh, $context->config('opachtdocs');
+        for ( grep { not /^\.|\.\.|lib$/ } readdir($fh) ) {
+            $dirs .= ' ' . "$opachtdocs/$_";
+        }
+        push @{$self->{interface}}, {
+            name   => "$_",
+            dir    => $dirs,
+            suffix => "-marc-$_.po",
+        };
+    }
+
     bless $self, $class;
 }
 
@@ -333,29 +348,37 @@ sub install_tmpl {
     my ($self, $files) = @_;
     say "Install templates" if $self->{verbose};
     for my $trans ( @{$self->{interface}} ) {
-        print
-            "  Install templates '$trans->{name}'\n",
-            "    From: $trans->{dir}/en/\n",
-            "    To  : $trans->{dir}/$self->{lang}\n",
-            "    With: $self->{path_po}/$self->{lang}$trans->{suffix}\n"
+        my @t_dirs = split(" ", $trans->{dir});
+        for my $t_dir ( @t_dirs ) {
+            my @files   = @$files;
+            my @nomarc = ();
+            print
+                "  Install templates '$trans->{name}'\n",
+                "    From: $t_dir/en/\n",
+                "    To  : $t_dir/$self->{lang}\n",
+                "    With: $self->{path_po}/$self->{lang}$trans->{suffix}\n"
                 if $self->{verbose};
 
-        my $trans_dir = ( $trans->{name} =~ /help/ )?"$trans->{dir}":"$trans->{dir}/en/";
-        my $lang_dir  = ( $trans->{name} =~ /help/ )?"$trans->{dir}":"$trans->{dir}/$self->{lang}";
-        $lang_dir =~ s|/en/|/$self->{lang}/|;
-        mkdir $lang_dir unless -d $lang_dir;
-        my $excludes  = ( $trans->{name} =~ /UI/   )?"-x 'help'":"";
-
-        system
-            "$self->{process} install " .
-            "-i $trans_dir " .
-            "-o $lang_dir  ".
-            "-s $self->{path_po}/$self->{lang}$trans->{suffix} -r $excludes" .
-            (
-                @$files
-                    ? ' -f ' . join ' -f ', @$files
-                    : ''
-            )
+            my $trans_dir = ( $trans->{name} =~ /help/ )?"$t_dir":"$t_dir/en/";
+            my $lang_dir  = ( $trans->{name} =~ /help/ )?"$t_dir":"$t_dir/$self->{lang}";
+            $lang_dir =~ s|/en/|/$self->{lang}/|;
+            mkdir $lang_dir unless -d $lang_dir;
+            my $excludes = ( $trans->{name} !~ /help/   )?"":"-x 'help'";
+            # if installing MARC po file, only touch corresponding files
+            my $marc     = ( $trans->{name} =~ /MARC/ )?"-m \"$trans->{name}\"":"";            # for MARC translations
+            # if not installing MARC po file, ignore all MARC files
+            @nomarc      = ( 'marc21', 'unimarc', 'normarc' ) if ( $trans->{name} !~ /MARC/ ); # hardcoded MARC variants
+
+            system
+                "$self->{process} install " .
+                "-i $trans_dir " .
+                "-o $lang_dir  ".
+                "-s $self->{path_po}/$self->{lang}$trans->{suffix} -r " .
+                "$excludes " .
+                "$marc " .
+                ( @files   ? ' -f ' . join ' -f ', @files : '') .
+                ( @nomarc  ? ' -n ' . join ' -n ', @nomarc : '');
+        }
     }
 }
 
@@ -365,25 +388,30 @@ sub update_tmpl {
 
     say "Update templates" if $self->{verbose};
     for my $trans ( @{$self->{interface}} ) {
+        my @files   = @$files;
+        my @nomarc = ();
         print
             "  Update templates '$trans->{name}'\n",
             "    From: $trans->{dir}/en/\n",
             "    To  : $self->{path_po}/$self->{lang}$trans->{suffix}\n"
                 if $self->{verbose};
-        my $lang_dir = "$trans->{dir}/$self->{lang}";
 
-        my $trans_dir = ( $trans->{name} =~ /help/ )?"$trans->{dir}":"$trans->{dir}/en/";
-        my $excludes  = ( $trans->{name} =~ /UI/   )?"-x 'help'":"";
+        my $trans_dir = ( $trans->{name} =~ /help/ )?"$trans->{dir}":join("/en/ -i ",split(" ",$trans->{dir}))."/en/"; # multiple source dirs
+        # do no process 'help' dirs unless needed
+        my $excludes  = ( $trans->{name} !~ /help/ )?"-x help":"";
+        # if processing MARC po file, only use corresponding files
+        my $marc      = ( $trans->{name} =~ /MARC/ )?"-m \"$trans->{name}\"":"";            # for MARC translations
+        # if not processing MARC po file, ignore all MARC files
+        @nomarc       = ( 'marc21', 'unimarc', 'normarc' ) if ( $trans->{name} !~ /MARC/ );      # hardcoded MARC variants
 
         system
             "$self->{process} update " .
             "-i $trans_dir " .
-            "-s $self->{path_po}/$self->{lang}$trans->{suffix} -r $excludes" .
-            (
-                @$files
-                    ? ' -f ' . join ' -f ', @$files
-                    : ''
-            )
+            "-s $self->{path_po}/$self->{lang}$trans->{suffix} -r " .
+            "$excludes " .
+            "$marc "     .
+            ( @files   ? ' -f ' . join ' -f ', @files : '') .
+            ( @nomarc  ? ' -n ' . join ' -n ', @nomarc : '');
     }
 }
 
@@ -405,24 +433,29 @@ sub create_tmpl {
 
     say "Create templates\n" if $self->{verbose};
     for my $trans ( @{$self->{interface}} ) {
+        my @files   = @$files;
+        my @nomarc = ();
         print
             "  Create templates .po files for '$trans->{name}'\n",
             "    From: $trans->{dir}/en/\n",
             "    To  : $self->{path_po}/$self->{lang}$trans->{suffix}\n"
                 if $self->{verbose};
 
-        my $trans_dir = ( $trans->{name} =~ /help/ )?"$trans->{dir}":"$trans->{dir}/en/";
-        my $excludes  = ( $trans->{name} =~ /UI/   )?"-x 'help'":"";
+        my $trans_dir = ( $trans->{name} =~ /help/ )?"$trans->{dir}":join("/en/ -i ",split(" ",$trans->{dir}))."/en/"; # multiple source dirs
+        my $excludes  = ( $trans->{name} !~ /help/ )?"-x help":"";
+        # if processing MARC po file, only use corresponding files
+        my $marc      = ( $trans->{name} =~ /MARC/ )?"-m \"$trans->{name}\"":"";            # for MARC translations
+        # if not processing MARC po file, ignore all MARC files
+        @nomarc       = ( 'marc21', 'unimarc', 'normarc' ) if ( $trans->{name} !~ /MARC/ ); # hardcoded MARC variants
 
         system
             "$self->{process} create " .
             "-i $trans_dir " .
-            "-s $self->{path_po}/$self->{lang}$trans->{suffix} -r $excludes" .
-            (
-                @$files
-                    ? ' -f ' . join ' -f ', @$files
-                    : ''
-            )
+            "-s $self->{path_po}/$self->{lang}$trans->{suffix} -r " .
+            "$excludes " .
+            "$marc " .
+            ( @files  ? ' -f ' . join ' -f ', @files   : '') .
+            ( @nomarc ? ' -n ' . join ' -n ', @nomarc : '');
     }
 }
 
@@ -584,8 +617,12 @@ appropriate directory.
 
 =item translate create F<lang>
 
-Create 3 .po files in F<po> subdirectory: (1) from opac pages templates, (2)
-intranet templates, and (3) from preferences.
+Create 4 kinds of .po files in F<po> subdirectory:
+(1) one from each theme on opac pages templates,
+(2) intranet templates and help,
+(3) preferences, and
+(4) one for each MARC dialect.
+
 
 =over
 
@@ -594,7 +631,7 @@ intranet templates, and (3) from preferences.
 Contains extracted text from english (en) OPAC templates found in
 <KOHA_ROOT>/koha-tmpl/opac-tmpl/{theme}/en/ directory.
 
-=item F<lang>-intranet.po
+=item F<lang>-staff-prog.po and F<lang>-staff-help.po
 
 Contains extracted text from english (en) intranet templates found in
 <KOHA_ROOT>/koha-tmpl/intranet-tmpl/prog/en/ directory.
@@ -605,6 +642,11 @@ Contains extracted text from english (en) preferences. They are found in files
 located in <KOHA_ROOT>/koha-tmpl/intranet-tmpl/prog/en/admin/preferences
 directory.
 
+=item F<lang>-marc-{MARC}.po
+
+Contains extracted text from english (en) files from opac and intranet,
+related with MARC dialects.
+
 =back
 
 =item pref-trans update F<lang>