FileDocCategorySizeDatePackage
SimpleTokenizer.javaAPI DocphoneME MR2 API (J2ME)3904Wed May 02 18:00:34 BST 2007com.sun.perseus.util

SimpleTokenizer

public final class SimpleTokenizer extends Object
This simple tokenizer is able to break down a String into tokens, given a single delimiter character.
version
$Id: SimpleTokenizer.java,v 1.3 2006/04/21 06:35:58 st125089 Exp $

Fields Summary
protected String
data
The data to parse.
protected int
length
The length of the string to parse.
protected char[]
del
The array of delimiters.
protected int
cur
The current parse index.
Constructors Summary
public SimpleTokenizer(String data, String delimiters)

param
data the string to tokenizer. Should not be null.
param
delimiters each character in the string is considered to be a delimiter. Should not be null.
return
an array of tokens.


                                                 
         
        if (data == null || delimiters == null) {
            throw new IllegalArgumentException();
        }

        this.data = data;
        this.length = data.length();
        del = delimiters.toCharArray();

        // Initialize by skipping delimiters.
        skipDelimiters();
Methods Summary
public intcountTokens()

return
the number of tokens

        int n = 0;
        int tmpCur = cur;

        cur = 0;
        while (cur < length) {
            skipDelimiters();
            if (cur < length) {
                n++;
            }
            skipToken();
        }

        cur = tmpCur;

        return n;
booleancurIsDelimiter()

return
true if the current character is a delimiter.

        char c = data.charAt(cur);
        for (int i = 0; i < del.length; i++) {
            if (c == del[i]) {
                return true;
            }
        }
        return false;
public booleanhasMoreTokens()

return
true if there are more tokens available.

        return cur < length;
public java.lang.StringnextToken()

returns
the nextToken

        if (!hasMoreTokens()) {
            return null;
        }

        // Now, build the new token.
        int s = cur;
        cur++;
        skipToken();
        int e = cur;

        // Skip all characters, starting at the current position, which
        // match one of the delimiters.
        skipDelimiters();

        return data.substring(s, e);
voidskipDelimiters()
Moves the current position to the first next character which is a delimiter.

        while (cur < length && curIsDelimiter()) {
            cur++;
        }
voidskipToken()
Moves the current position to the first next character which is not a delimiter.

        while (cur < length && !curIsDelimiter()) {
            cur++;
        }