GeneralString, UniversalString, or UTF8String?

[Ivan Shmakov]

        I'm writing an application which has to support the full range
        of Unicode characters being stored to and retrieved from its
        (structured) data files, which are to be binary for both space
        and performance reasons.

        Although my experience with ASN.1 is limited, it seems like a
        suitable base format for such an application.  As per [1], my
        guess is that I should use either UniversalString or UTF8String
        to store Unicode text strings (though I don't understand the
        difference.)  However, as per [2], the only character sequence
        type currently supported by GNU Libtasn1 is GeneralString.

        OTOH, the Perl part of the my application is likely to be based
        on Convert::ASN1 [3], which supports both UniversalString and
        UTF8String, although it seems to encode both of them identically
        (sans the type tag.)  Which makes me wonder: whether should I
        use GeneralString, or request for the support for
        UniversalString (or UTF8String) to be added to Libtasn1?

        TIA.

[1] http://www.obj-sys.com/asn1tutorial/node128.html
[2] http://www.gnu.org/software/libtasn1/manual/libtasn1.html
[3] http://search.cpan.org/perldoc?Convert::ASN1

        To note is that GeneralString is also supported by
        Convert::ASN1.  The test ASN.1 definition I've used is:

$ cat < example.asn1 
Example { 99 99 99 99 }

DEFINITIONS EXPLICIT TAGS ::=

BEGIN

Foo ::= SEQUENCE {
    foo     INTEGER,
    bar     GeneralString
}

END
$ 

        And the test itself:

$ asn1Coding \
      -o >(perl -we 'use strict;
                     require Convert::ASN1;
                     require Data::Dump;
                     require IO::Handle;
                     my $asn
                         = Convert::ASN1->new ();
                     $asn->prepare (q {
                             SEQUENCE {
                                 foo INTEGER,
                                 bar GeneralString
                             }
                         })
                         or die ();
                     my $in = \*STDIN;
                     binmode ($in);
                     local $/
                         = undef;
                     my $buf
                         = <$in>;
                     my $dec
                         = $asn->decode ($buf);
                     print ("XXX: ", Data::Dump::dump ($dec, $buf), "\n");
                     ') \
      example.asn1 \
      <(printf %s\\n 'x Example.Foo' 'foo 3' 'bar x') 
Parse: done.

var=x, value=Example.Foo
var=foo, value=3
var=bar, value=x

name:NULL  type:SEQUENCE
  name:foo  type:INTEGER  value:0x03
  name:bar  type:GENERALSTRING  value:78

Coding: SUCCESS

-----------------
Number of bytes=8
30 06 02 01 03 1b 01 78
-----------------

OutputFile=/dev/fd/63

Writing: done.
XXX: ({ bar => "x", foo => 3 }, "0\6\2\1\3\e\1x")
$ 

-- 
FSF associate member #7257