FileDocCategorySizeDatePackage
ReaderTokenizer.javaAPI DocExample3988Sat Jan 24 10:44:26 GMT 2004je3.io

ReaderTokenizer

public class ReaderTokenizer extends je3.classes.AbstractTokenizer
This Tokenizer implementation extends AbstractTokenizer to tokenize a stream of text read from a java.io.Reader. It implements the createBuffer() and fillBuffer() methods required by AbstractTokenizer. See that class for details on how these methods must behave. Note that a buffer size may be selected, and that this buffer size also determines the maximum token length. The Test class is a simple test that tokenizes a file and uses the tokens to produce a copy of the file

Fields Summary
Reader
in
Constructors Summary
public ReaderTokenizer(Reader in)

 this(in, 16*1024);
public ReaderTokenizer(Reader in, int bufferSize)

	this.in = in;  // Remember the reader to read input from
	// Tell our superclass about the selected buffer size.
	// The superclass will pass this number to createBuffer()
	maximumTokenLength(bufferSize);
Methods Summary
protected voidcreateBuffer(int bufferSize)

	// Make sure AbstractTokenizer only calls this method once
	assert text == null;
	this.text = new char[bufferSize];  // the new buffer
	this.numChars = 0;                 // how much text it contains
protected booleanfillBuffer()

	// Make sure AbstractTokenizer is upholding its end of the bargain
	assert text!=null && 0 <= tokenStart && tokenStart <= tokenEnd &&
	    tokenEnd <= p && p <= numChars && numChars <= text.length;

	// First, shift already tokenized characters out of the buffer
	if (tokenStart > 0) {
	    // Shift array contents
	    System.arraycopy(text, tokenStart, text, 0, numChars-tokenStart);
	    // And update buffer indexes
	    tokenEnd -= tokenStart; 
	    p -= tokenStart;
	    numChars -= tokenStart;
	    tokenStart = 0; 
	}

	// Now try to read more characters into the buffer
	int numread = in.read(text, numChars, text.length-numChars);
	// If there are no more characters, return false
	if (numread == -1) return false;
	// Otherwise, adjust the number of valid characters in the buffer
	numChars += numread;
	return true;