Java - StreamTokenizer class

---

---

Introduction

The Java StreamTokenizer class takes an input stream and parses it into "tokens", allowing the tokens to be read one at a time. The stream tokenizer can recognize identifiers, numbers, quoted strings, and various comment styles.

Class declaration

Following is the declaration for Java.io.RandomAccessFile class −

public class StreamTokenizer
   extends Object

Field

Following are the fields for Java.io.StreamTokenizer class −

• double nval − If the current token is a number, this field contains the value of that number.

• String sval − If the current token is a word token, this field contains a string giving the characters of the word token.

• static int TT_EOF − A constant indicating that the end of the stream has been read.

• static int TT_EOL − A constant indicating that the end of the line has been read.

• static int TT_NUMBER − A constant indicating that a number token has been read.

• static int TT_WORD − A constant indicating that a word token has been read.

• int ttype − After a call to the nextToken method, this field contains the type of the token just read.

Class constructors

Class methods

Methods inherited

This class inherits methods from the following classes −

• Java.io.Object

Example - Use # as a comment character

The following example shows the usage of StreamTokenizer commentChar(int ch) method.

StreamTokenizerDemo.java

package com.tutorialspoint;

import java.io.IOException;
import java.io.Reader;
import java.io.StreamTokenizer;
import java.io.StringReader;

public class StreamTokenizerDemo {
   public static void main(String[] args) throws IOException {
      String input = "Java 101 # This is a comment\nPython 202";

      Reader reader = new StringReader(input);
      StreamTokenizer tokenizer = new StreamTokenizer(reader);

      tokenizer.commentChar('#'); // Treat '#' as start of comment

      System.out.println("Tokens:");
      while (tokenizer.nextToken() != StreamTokenizer.TT_EOF) {
         if (tokenizer.ttype == StreamTokenizer.TT_WORD) {
            System.out.println("Word: " + tokenizer.sval);
         } else if (tokenizer.ttype == StreamTokenizer.TT_NUMBER) {
            System.out.println("Number: " + tokenizer.nval);
         }
      }
   }
}

Output

Let us compile and run the above program, this will produce the following result−

Tokens:
Word: Java
Number: 101.0
Word: Python
Number: 202.0

Explanation

• Input includes # This is a comment, which gets skipped.

• Only Java 101 and Python 202 are parsed as tokens.

Example - eolIsSignificant(true) – Detect end of lines

The following example shows the usage of StreamTokenizer eolIsSignificant(boolean flag) method.

StreamTokenizerDemo.java

package com.tutorialspoint;

import java.io.IOException;
import java.io.Reader;
import java.io.StreamTokenizer;
import java.io.StringReader;

public class StreamTokenizerDemo {
   public static void main(String[] args) throws IOException {
      String input = "Hello 123\nWorld 456\nDone";

      Reader reader = new StringReader(input);
      StreamTokenizer tokenizer = new StreamTokenizer(reader);

      tokenizer.eolIsSignificant(true); // Enable EOL tokens

      System.out.println("Tokens:");
      while (tokenizer.nextToken() != StreamTokenizer.TT_EOF) {
         switch (tokenizer.ttype) {
            case StreamTokenizer.TT_WORD:
               System.out.println("Word: " + tokenizer.sval);
               break;
            case StreamTokenizer.TT_NUMBER:
               System.out.println("Number: " + tokenizer.nval);
               break;
            case StreamTokenizer.TT_EOL:
               System.out.println("End of line");
               break;
         }
      }
   }
}

Output

Let us compile and run the above program, this will produce the following result−

Tokens:
Word: Hello
Number: 123.0
End of line
Word: World
Number: 456.0
End of line
Word: Done

Explanation

• With eolIsSignificant(true), the tokenizer reports each line break.

• You'll see "End of line" printed after each line of tokens.

Example - Print line number for each token

The following example shows the usage of StreamTokenizer lineno() method.

StreamTokenizerDemo.java

package com.tutorialspoint;

import java.io.IOException;
import java.io.Reader;
import java.io.StreamTokenizer;
import java.io.StringReader;

public class StreamTokenizerDemo {
   public static void main(String[] args) throws IOException {
      String input = "Java 101\nPython 202\nC 303";

      Reader reader = new StringReader(input);
      StreamTokenizer tokenizer = new StreamTokenizer(reader);

      tokenizer.eolIsSignificant(true); // Count lines accurately

      System.out.println("Tokens with line numbers:");
      while (tokenizer.nextToken() != StreamTokenizer.TT_EOF) {
         if (tokenizer.ttype == StreamTokenizer.TT_WORD) {
            System.out.println("Line " + tokenizer.lineno() + ": Word - " + tokenizer.sval);
         } else if (tokenizer.ttype == StreamTokenizer.TT_NUMBER) {
            System.out.println("Line " + tokenizer.lineno() + ": Number - " + tokenizer.nval);
         } else if (tokenizer.ttype == StreamTokenizer.TT_EOL) {
            // Line numbers increase after TT_EOL
         }
      }
   }
}

Output

Let us compile and run the above program, this will produce the following result−

Tokens with line numbers:
Line 1: Word - Java
Line 1: Number - 101.0
Line 2: Word - Python
Line 2: Number - 202.0
Line 3: Word - C
Line 3: Number - 303.0

Explanation

• Each token is printed along with the line number.

• lineno() reflects the line the token was found on.