Making Tokens of a Java Source Code

After reading this lesson, you will be able to break java program into tokens.

Making Tokens of a Java Source Code


If you are new in Java programming then our tutorials and examples will be helpful in understanding Java programming in the most simplest way. Here after reading this lesson, you will be able to break java program into tokens. 

The use of  token is just like as we take tokens for getting a milk or get into a metro, so that the one token can be read at a time. 
Suppose if we have to make a token of a java file then we will do. Firstly we have to make a class TokenizingJavaSourceCode. Declare a main method in it . Now make a object of BufferedReader class and pass the InputStream class object into it which will read the name of the java file at run time. Now we need to read the file so we use FileReader class ,after reading a file we need to generate a tokens out of that file so, we use class StreamTokenizer and pass reference of a Reader class in its constructor. In this program we are using one method of StreamTokenizer class. 

To solve this problem we have make use of the following classes and methods.

InputStream: It is a abstract class and the parent class of all the Stream classes.

BufferedReader: This class reads the text from the input stream. This class extends Reader class. 

File: It is a class implementing Serializable and Comparable interface.

FileReader:  This class  reads the file in the character format. It extends InputStreamReader.

StreamTokenizer: This class takes a input stream and break it into tokens so that one token can be read at a time. It takes FileReader instance as a input in its constructor. 

readLine(): It is a method of BufferedReader class which reads a line.

exists(): It will check whether the file exists or not.

nextToken(): It gives the next token.

We have also used one final static variable TT_EOF which tells the reader that the end of the file has been read.

The code of this program is given below:

import java.util.*;

public class TokenizingJavaSourceCode{
  //public static final int TT_EOL; 
  public static void main(String[] argsthrows IOException{
  BufferedReader in = new BufferedReader(new InputStreamReader(;
  System.out.print("Please enter a java file name: ");
  String filename = in.readLine();
  System.out.println("This is not a java file.");
  File javaFile = new File(filename);
  FileReader file = new FileReader(filename);
  StreamTokenizer streamTokenizer = new StreamTokenizer(file);
  // It will go through the file and 
  //gives the number of tokens in the file

  int i=0;
  int numberOfTokensGenerated = 0;
  while(i != StreamTokenizer.TT_EOF){
  i = streamTokenizer.nextToken();
  System.out.println("Number of tokens = " + numberOfTokensGenerated);
  System.out.println("File does not exist!");

Output of this program is given below:

C:\Java Tutorial>java TokenizingJavaSourceCode
Please enter a java file name:
Number of tokens = 158

C:\Java Tutorial>

Download this program

Share on Google+Share on Google+

Making Tokens of a Java Source Code

Posted on: June 11, 2007 If you enjoyed this post then why not add us on Google+? Add us to your Circles



Discuss: Making Tokens of a Java Source Code  

Post your Comment

Your Name (*) :
Your Email :
Subject (*):
Your Comment (*):
  Reload Image
July 2, 2011

I can't believe I've been going for years without konnwig that.
June 12, 2013

Woot, I will cetrinaly put this to good use!
September 25, 2011
what is token in java?

give me the solution immediately
Ahmed Elmasry
December 31, 2011
how we know types of tokens

how we know types of tokens:_ _keyword _identifier _number _real number and .........
sara parsa
December 30, 2012
tokenizing a java source code

I need some guidance to be able to tokenize a java sourcecode into operators,keywords and variable.Please help Thanks