[Koha-devel] Re: MARC character encoding

[Ed Summers]

Hi Paul:

On Wed, Jan 15, 2003 at 03:20:47PM +0100, paul POULAIN wrote:
> Could someone explain how to translate the "MARC21" charset to a more 
> convenient one (and which is more convenient ?)
> Same question for UNIMARC (which is ISO646 if my docs are right)

If we lived in a perfect world we would all be using Unicode (UTF8)
since it covers so many of the worlds scripts [1]. Unfortunately the
world is not perfect. MARC has been around longer than Unicode, so 
MARC-8 character encoding to allow non Latin scripts to live in MARC 
records. I guess the world has bigger problems than character encodings
(Mr George Bush comes to mind), but I'll leave that particular problem
alone :)

I wasn't aware that UNIMARC had defined a different standard for
character encoding. Isn't ISO646 just an synonym for ASCII? [2] Which docs 
describe the character sets used in UNIMARC?

> I tried MARC-Charset, which seems to translate from "MARC21" to UNICODE, 
> but i don't know what to do with my unicode ;-(

Yes, MARC::Charset is an implementation of the MARC-8 ==> Unicode
(UTF-8) mappings published by the Library of Congress. [3] In MARC-8
there is a special way of 'escaping' to other character sets (Hebrew,
Cyrillic, East-Asian, etc). 

> I tried :
>    my $charset = MARC::Charset->new();
>    print $charset->to_utf8($unimarc->as_formatted())."\n";
> where $unimarc is a MARC::Record containing a MARC21 record converted to 
> UNIMARC (hope everybody understand this : i mapped marc21 fields to 
> unimarc ones)

The to_utf8() method will take a string of characters (encoded in
MARC-8) and convert them to Unicdoe (UTF8). Initially I wanted to do
this so that MARC records could be expressed as XML with UTF8 encoding.

You mapped all the UNIMARC fields to MARC fields!?! I was under the
impression that this was quite a big undertaking to do completely. Is
your code currently checked into CVS? Having a UNIMARC filter in
MARC::Record (MARC::File::UNIMARC) has been a long term goal. Maybe we
could roll this work into the MARC::Record package?

I'm not sure how you want to handle character encoding in Koha. Does
MySQL properly handle Unicode (UTF8)? If it does I think long term
Koha should probably attempt to store everything in UTF8. Luckily UTF8
is backwards compatible with plain vanilla ASCII. Even if MySQL handles
storing UTF8, it would take some research to make sure DBD::mysql also
does.

Otherwise, there needs to be some global config option that determines
Koha's character set encoding. There is the Encode [4] module (standard
with 5.8.0) which could handle translating between a variety of character
encodings.

Hope this helps more than it hurts :)

//Ed

[1] http://www.unicode.org
[2] http://czyborra.com/charsets/iso646.html
[3] http://www.loc.gov/marc/specifications/specchartables.html
[4] http://search.cpan.org/author/DANKOGAI/Encode-1.84/

-- 

% perl -MData::Dumper -e "print Dumper($me)"
$VAR1 = {
          'WEB' => 'http://www.inkdroid.org',
          'NAME' => 'Ed Summers',
          'AIM' => 'inkdroid',
          'EMAIL' => 'address@hidden'
        };

pgpaIn4aZRTDs.pgp
Description: PGP signature