FileDocCategorySizeDatePackage
TokenBuffer.javaAPI DocGlassfish v2 API3693Wed Aug 30 15:34:10 BST 2006persistence.antlr

TokenBuffer

public class TokenBuffer extends Object
A Stream of Token objects fed to the parser from a Tokenizer that can be rewound via mark()/rewind() methods.

A dynamic array is used to buffer up all the input tokens. Normally, "k" tokens are stored in the buffer. More tokens may be stored during guess mode (testing syntactic predicate), or when LT(i>k) is referenced. Consumption of tokens is deferred. In other words, reading the next token is not done by conume(), but deferred until needed by LA or LT.

see
persistence.antlr.Token
see
persistence.antlr.Tokenizer
see
persistence.antlr.TokenQueue

Fields Summary
protected TokenStream
input
int
nMarkers
int
markerOffset
int
numToConsume
TokenQueue
queue
Constructors Summary
public TokenBuffer(TokenStream input_)
Create a token buffer


         
       
        input = input_;
        queue = new TokenQueue(1);
Methods Summary
public final intLA(int i)
Get a lookahead token value

        fill(i);
        return queue.elementAt(markerOffset + i - 1).type;
public final persistence.antlr.TokenLT(int i)
Get a lookahead token

        fill(i);
        return queue.elementAt(markerOffset + i - 1);
public final voidconsume()
Mark another token for deferred consumption

        numToConsume++;
private final voidfill(int amount)
Ensure that the token buffer is sufficiently full

        syncConsume();
        // Fill the buffer sufficiently to hold needed tokens
        while (queue.nbrEntries < amount + markerOffset) {
            // Append the next token
            queue.append(input.nextToken());
        }
public persistence.antlr.TokenStreamgetInput()
return the Tokenizer (needed by ParseView)

        return input;
public final intmark()
Return an integer marker that can be used to rewind the buffer to its current state.

        syncConsume();
//System.out.println("Marking at " + markerOffset);
//try { for (int i = 1; i <= 2; i++) { System.out.println("LA("+i+")=="+LT(i).getText()); } } catch (ScannerException e) {}
        nMarkers++;
        return markerOffset;
public final voidreset()
Reset the input buffer to empty state

        nMarkers = 0;
        markerOffset = 0;
        numToConsume = 0;
        queue.reset();
public final voidrewind(int mark)
Rewind the token buffer to a marker.

param
mark Marker returned previously from mark()

        syncConsume();
        markerOffset = mark;
        nMarkers--;
//System.out.println("Rewinding to " + mark);
//try { for (int i = 1; i <= 2; i++) { System.out.println("LA("+i+")=="+LT(i).getText()); } } catch (ScannerException e) {}
private final voidsyncConsume()
Sync up deferred consumption

        while (numToConsume > 0) {
            if (nMarkers > 0) {
                // guess mode -- leave leading tokens and bump offset.
                markerOffset++;
            }
            else {
                // normal mode -- remove first token
                queue.removeFirst();
            }
            numToConsume--;
        }