Share on Google+Share on Google+

Making Tokens of a String

Tokens can be used where we want to break an application into tokens.

Making Tokens of a String


Tokens can be used where we want to break an application into tokens. We have to break a String  into tokens as well as we will know how many tokens has been generated. That's what we are trying to do in this program. In the program a string is passed into a constructor of StringTokenizer class. StringTokenizer is a class in java.util.package. We are using while loop to generate tokens. The following methods have been used in this program.

countTokens(): It gives the number of tokens remaining in the string.

hasMoreTokens(): It gives true if more tokens are available, else false.

nextToken(): It gives the next token available in the string.

To break a string into tokens what we need first is to create a class named StringTokenizing. Inside this class we will declare our main method. Make an object of class StringTokenizer and pass one string inside the constructor which you want to break into tokens. By using the instance of StringTokenizer call the method countTokens() which gives the number of tokens remaining in the string. It is a method of  StringTokenizer class. If the object have more tokens available then it will call method hasMoreTokens() and print the tokens by using nextToken().

The code of the program is given below:

import java.util.*;

public class StringTokenizing{
  public static void main(String[] args) {
  StringTokenizer stringTokenizer = new 
("You are tokenizing a string");
  System.out.println("The total no. of tokens 
   generated :  " 
+ stringTokenizer.countTokens());

Output of this program is given below:

C:\java>java StringTokenizing
The total no. of tokens generated : 5

Download this program:


Posted on: June 7, 2007 If you enjoyed this post then why not add us on Google+? Add us to your Circles

Share this Tutorial Follow us on Twitter, or add us on Facebook or Google Plus to keep you updated with the recent trends of Java and other open source platforms.

Discuss: Making Tokens of a String   View All Comments

Post your Comment

Your Name (*) :
Your Email :
Subject (*):
Your Comment (*):
  Reload Image
September 22, 2011

I don't know who you wrote this for but you hepeld a brother out.
July 8, 2012
finding frequency of tokens

after tokanizing,how to find the frequency of the tokens that have appeared and then sort the tokens according to its frequency.......
November 26, 2012
Splitting a string

how can we split 12+12 into three tokens like 12,+ and 12..please reply me if any alternate method available..