Re: hyphenating non-english characters

[G. Branden Robinson]

Hi Gáspár,

At 2024-07-27T08:51:32+0200, Gáspár Gergő wrote:
> I'm trying to make justified text look nicer, so I turned to
> hyphenation.

A reasonable desire.

> Hungarian is not supported out of the box by groff, but I found a tex
> patterns file which seems quite good, that is what I tried to use, to
> not much success.

You didn't indicate where the hyphenation pattern file came from
exactly (though its name is suggestive), or attach it, but odds are it's
UTF-8 encoded, and that is a problem for GNU troff in its present state,
which supports only single-byte encodings.  Further, in groff 1.24,
support for the ASCII-incompatible CCSID ["code page"] 1047 will be
withdrawn.  (This matters because UTF-8 is a compatible extension of ISO
646 a.k.a. "ASCII" but not of CCISD 1047.)

https://savannah.gnu.org/bugs/?65724

The good news is that if you convert the hyphenation pattern file to ISO
Latin-2 (ISO 8859-2), you should be able to use it.

> Hyphenation happens, but not as often as I'd hope.  After some more
> reading, I think the problem might be with the accented Hungarian
> characters not having hyphenation codes assigned to them, since
> hyphenation seemingly only happens near non-accented vowels.

That seems likely to me.  The code that reads hyphenation patterns
simply does not expect a multi-byte character encoding.

https://git.savannah.gnu.org/cgit/groff.git/tree/src/roff/troff/env.cpp?h=1.23.0#n3856

> These are the requests that I used for hyphenation originally:
> .hla hu
> .hpf /home/gergo/projekt/jella/huhyphn.tex
> .hy 1

This is good, but incomplete, setup.  As you noted below, `hcode`
requests to assign hyphenation codes to non-ASCII code points are also
necessary.

It will also help to tell GNU troff to use an accurate input character
mapping.

.mso latin2.tmac

I would insert that request before the `hpf` request, and invoke `hcode`
requests _before_ `hpf`, but I will admit I haven't experimented with
arbitrary arrangements of these to see what breaks hyphenation support.

I will say with some confidence that `hla` should come first, because
the hyphenation language is a property of the environment, and is the
way you tell GNU troff which language you'll be defining hyphenation
codes and patterns for.

Due to that property and because you can change environments, it's not
difficult to support multilingual documents that apply appropriate
hyphenation rules to the input language.

> The manual told me that
>
> > A hyphenation code must be an ordinary character (not a special
> > character escape sequence) other than a digit or a space.

Yes.  In groff 1.24, loading "latin2.tmac" is what will _make_ input
characters like ű "ordinary" (otherwise they're out of range and get
warned about).

$ ~/groff-stable/bin/groff --version | head -n 1
GNU groff version 1.23.0
$ printf '\370\n' | ~/groff-stable/bin/groff -a -winput
<beginning of page>
</o>
$ groff --version | head -n 1
GNU groff version 1.23.0.1589-aa44
$ printf '\370\n' | groff -a -winput<beginning of page>
troff:<standard input>:1: warning: character with input code 248 not defined

I haven't decided yet but 1.24's default might be to _not_ load
"latin1.tmac", since English is the default language and most English
doesn't need Latin-1 characters, but more importantly to get users into
position for a conversion to UTF-8 support in, I hope, groff 1.25.  I
don't think there is time given current developer resources to get groff
to UTF-8 from end to end in one fell swoop in time for a release this
year, and also, assuming that one is going from an 8-bit character
encoding to UTF-8 leads to mojibake.  Because UTF-8 is compatible with
7-bit ASCII, if groff 1.24 gets people to either (1) use ASCII or (2)
make their documents (or their system, via "troffrc" or similar)
_declare the default encoding they expect_, we can avoid frustrating
them later.

See <https://savannah.gnu.org/bugs/?65955>.

> So I tried following the example, with these requests:
> .hcode á á Á á
> .hcode é é É é
> .hcode í í Í í
> .hcode ó ó Ó ó
> .hcode ö ö Ö ö
> .hcode ő ő Ő ő
> .hcode ú ú Ú ú
> .hcode ü ü Ü ü
> .hcode ű ű Ű ű

I expect that to work if you (1) get the hyphenation patterns recoded to
Latin-2 and (2) load "latin2.tmac" before attempting to format text.

> However, groff throws errors saying "error: hyphenation code must be
> ordinary character".

If you're using groff 1.23 or earlier, I'm a little surprised by this.

If you're using groff Git's master branch of recent vintage, this is
what I would expect.

> I tried with and without preconv to no avail.

preconv(1) is important in general for non-English language support in
groff, but does not apply to this situation because hyphenation pattern
files are parsed independently of text formatting and input processing.

> The example supplied in the manual, with German characters, didn't
> work either.

That is also a surprise, if you're running groff 1.23.  I will attempt
to reproduce this.  In Git, it's not a surprise and is a consequence of
me not getting around to updating the documentation yet once I started
down the road of resolving Savannah ticket #65724.

It _should_ work if you load the German localization file.

.mso de.tmac

Or "groff -mde".  But that will also perform the `hcode` requests for
you.

> What could be the problem here?

I hope the foregoing has shed some light.

If you resolve these issues you'll be well along the path to being able
to contribute Hungarian language support to groff, which I would be
interested to integrate.  The main items remaining would, I think, be:

1.  Select appropriate automatic hyphenation modes (these are dictated
    by the TeX hyphenation patterns).

    https://git.savannah.gnu.org/cgit/groff.git/tree/tmac/LOCALIZATION

2.  Translate some stock words and phrases used by our macro packages.

    Here are the German ones, for example.

    https://git.savannah.gnu.org/cgit/groff.git/tree/tmac/de.tmac?h=1.23.0#n39

Regards,
Branden

signature.asc
Description: PGP signature