Making Tokens of a Java Source Code
This section of I/O tutorial will be helpful to
understand Java programming in the most
simplest and effecient way.
In Java, The StreamTokenizer class is used for
simple parsing of a Java source file into tokens. This class takes an input
stream, breaks up the ASCII text in a file it and parses it into
"tokens", allowing the tokens to be read one at a time. The parsing
process is controlled by a table and a number of flags that can be set to
various states. This class is extended from the java.io and java.util package.
The use of token is just as we take tokens
for getting a milk or to enter in metro, so that the one token can be read at a
time.
Lets understand the example shown below:
In the given program, firstly we make a class TokenizingJavaSourceCode.
Now make a object of BufferedReader class
and pass the InputStream class object into it which will read the name of
the java file at run time. Now we need to read the file so that we can use FileReader class
,After reading a file we need to generate a tokens
out of that file so that we can use the class StreamTokenizer and pass
the reference of a Reader
class in its constructor. In this program we are using one method of StreamTokenizer
class.
To solve this problem we have make use of the following
classes and methods.
InputStream: It is a abstract class and the
parent class of all the Stream classes.
BufferedReader: This class reads the text from
the input stream. This class extends Reader class.
File: It is a class implementing Serializable
and Comparable interface.
FileReader: This class reads the
file in the character format. It extends InputStreamReader.
StreamTokenizer: This class takes a input stream
and break it into tokens so that one token can be read at a time. It takes FileReader
instance as a input in its constructor.
readLine(): It is a method of BufferedReader class
which reads a line.
exists(): It will check whether the file exists
or not.
nextToken(): It gives the next token.
We have also used one final static variable TT_EOF
which tells the reader that the end of the file has been read.
The code of this program is given below:
import java.io.*;
import java.util.*;
public class TokenizingJavaSourceCode{
//public static final int TT_EOL;
public static void main(String[] args) throws IOException{
BufferedReader in = new BufferedReader(new InputStreamReader(System.in));
System.out.print("Please enter a java file name: ");
String filename = in.readLine();
if(!filename.endsWith(".java")){
System.out.println("This is not a java file.");
System.exit(0);
}
File javaFile = new File(filename);
if(javaFile.exists()){
FileReader file = new FileReader(filename);
StreamTokenizer streamTokenizer = new StreamTokenizer(file);
// It will go through the file and gives the number of tokens in the file
int i=0;
int numberOfTokensGenerated = 0;
while(i != StreamTokenizer.TT_EOF){
i = streamTokenizer.nextToken();
numberOfTokensGenerated++;
}
System.out.println("Number of tokens = " + numberOfTokensGenerated);
}
else{
System.out.println("File does not exist!");
System.exit(0);
}
}
}
|
Output of the program:
C:\Java Tutorial>java TokenizingJavaSourceCode
Please enter a java file name: TokenizingJavaSourceCode.java
Number of tokens = 158
C:\Java Tutorial>_ |
Download
this program
|
Del.icio.us
Digg
Google
Spurl
Blink
Furl
Simpy
Y! MyWeb
