Tokens can be used where we want to break an application into tokens.Tokens can be used where we want to break an application into tokens.
Tokens can be used where we want to break an application into tokens. We have to break a String into tokens as well as we will know how many tokens has been generated. That's what we are trying to do in this program. In the program a string is passed into a constructor of StringTokenizer class. StringTokenizer is a class in java.util.package. We are using while loop to generate tokens. The following methods have been used in this program.
countTokens(): It gives the number of tokens remaining in the string.
hasMoreTokens(): It gives true if more tokens are available, else false.
nextToken(): It gives the next token available in the string.
To break a string into tokens what we need first is to create a class named StringTokenizing. Inside this class we will declare our main method. Make an object of class StringTokenizer and pass one string inside the constructor which you want to break into tokens. By using the instance of StringTokenizer call the method countTokens() which gives the number of tokens remaining in the string. It is a method of StringTokenizer class. If the object have more tokens available then it will call method hasMoreTokens() and print the tokens by using nextToken().
The code of the program is given below:
Output of this program is given below:
The total no. of tokens generated : 5