Re: [Texmacs-dev] encoding conversion update

[Joris van der Hoeven]

Hi Felix,

> I finally got round to (virtually) finish my work on some c++ classes
> that allow one to "apply" a dictionary to a string in an efficient
> manner. You can have a look at them at
> 
> http://www.fbreuer.de/texmacs
> 
> Originally I wanted to put up a standalone tool that could apply a .scm
> file to a given text file but I kind of lost my struggle with the
> linker: I have no idea which set of .o files I have to link against to
> produce a working binary.

Thanks for this good work. I took a quick look at what you have done.
I think that it is good to incorporate hashtree's into TeXmacs,
since that is a clean data structure. The concatenators are not really
necessary, since all TeXmacs arrays can already be used in that way:
in fact, an array always has 2^p elements allocated in memory,
even if it uses less ones. This allows quick concatenation.
I still have to take a more detailed look to your converter class,
but it seels OK for inclusion too.

In the future, we will also need a routine which does the following:
given a string in a certain encoding and a list of cursor positions,
produce the converted string and the list of converted positions.
Of course, it is also important to have the file loader working.
It might also be good if you consider a converter as a resource
(see src/Resources).

> To continue work, it would be necessary to integrate the code with
> TeXmacs and the HTML export routines, so that one can actually export a
> file and see whether the encoding has been properly converted. I do not
> feel up to this integration-task tough. Joris and David, it would be
> nice if you could have a look at the code and tell me where (and if) it
> might be of any use.

David is working on the Html converter at this very moment.
When he we will have finished, I will try to merge everything together.
In the meanwhile, you might remove the concatenators and
finish the support for loading converters and let them become resources.
You may then add a test routine which takes three strings on input
(string to convert, input encoding, output encoding) and
which returns the converted string. This routine may be added
to the scheme interface.

I am also wondering about the efficiency of the hashtree method.
In the case when characters are directly mapped to characters
it might be better to do table lookup. But this is not urgent.

Thanks again for your great contribution,

Joris