[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [pdf-devel] Re: Modifications on pdf_token_read to get token boundar
From: |
David Vazquez |
Subject: |
Re: [pdf-devel] Re: Modifications on pdf_token_read to get token boundaries |
Date: |
Tue, 26 May 2009 03:18:02 +0200 |
User-agent: |
Gnus/5.11 (Gnus v5.11) Emacs/22.2 (gnu/linux) |
Michael Gold <address@hidden> writes:
> On Tue, May 26, 2009 at 00:53:46 +0200, address@hidden wrote:
> ...
>> We need to be able to determine the boundaries of a token that has
>> been read, for error reporting. We cannot rely on the stm used by the
>> token reader to determine the beginning position of a read token,
>> since it is skipping white characters.
>
> This behaviour could be changed by
> - adding a flag that causes token_read to return whitespace as a token;
> or,
> - adding a function/flag to advance to the beginning of the next token
>
It is preferable it is not work of the tokeniser module, I think.
Adding a new function for this seems nice for me. Although we could
simply the API if `pdf_token_reader_new' function consumes characters
until the first token, and `pdf_token_read' does same thing after of
read each token, then we could assume the token always is at the
current position of the stream, therefore we could use the
`pdf_stm_tell' function to get the beginning of each token.
Finally, I think we will not need the ending offset of a token.