StringTokenizer in Java

Java Tutorial

Java Tutorial Java Exercises Data Structures

The processing of text often consists of parsing a formatted input string. Parsing is the division of text into a set of discrete parts, or tokens, which in a certain sequence can convey a semantic meaning. The StringTokenizer class provides the first step in this parsing process, often called the lexer (lexical analyzer) or scanner. StringTokenizer implements the Enumeration interface. Therefore, given an input string, you can enumerate the individual tokens contained in it using StringTokenizer.

To use StringTokenizer, you specify an input string and a string that contains delimiters. Delimiters are characters that separate tokens. Each character in the delimiters string is considered a valid delimiter—for example, ",;:" sets the delimiters to a comma, semicolon, and colon. The default set of delimiters consists of the whitespace characters: space, tab, newline, and carriage return.

The StringTokenizer constructors are shown here:

StringTokenizer(String str) 
StringTokenizer(String str, String delimiters) 
StringTokenizer(String str, String delimiters, boolean delimAsToken)

In all versions, str is the string that will be tokenized. In the first version, the default delimiters are used. In the second and third versions, delimiters is a string that specifies the delimiters. In the third version, if delimAsToken is true, then the delimiters are also returned as tokens when the string is parsed. Otherwise, the delimiters are not returned.

Delimiters are not returned as tokens by the first two forms. Once you have created a StringTokenizer object, the nextToken( ) method is used to extract consecutive tokens. The hasMoreTokens( ) method returns true while there are more tokens to be extracted. Since StringTokenizer implements Enumeration, the hasMoreElements( ) and nextElement( ) methods are also implemented, and they act the same as hasMoreTokens( ) and nextToken( ), respectively.

Here is an example that creates a StringTokenizer to parse "key=value" pairs. Consecutive sets of "key=value" pairs are separated by a semicolon.



import java.util.StringTokenizer; 
class Test1
 { 
static String in = "title=C & Java-Samples;" + 
"author=Rama;" + 
"publisher=cprogramcoding.com;" + 
"copyright=2018;"; 
public static void main(String args[]) { 
StringTokenizer st = new StringTokenizer(in, "=;"); 
while(st.hasMoreTokens()) { 
String key = st.nextToken(); 
String val = st.nextToken(); 
System.out.println(key + "\t" + val); 
} 
} 
}


Output:


title   C & Java-Samples
author  Rama
publisher       cprogramcoding.com
copyright       2018

The java.util.StringTokenizer class allows you to break a string into tokens. It is simple way to break string.

It doesn't provide the facility to differentiate numbers, quoted strings, identifiers etc. like StreamTokenizer class. We will discuss about the StreamTokenizer class in I/O chapter.

Constructors of StringTokenizer class

There are 3 constructors defined in the StringTokenizer class.

ConstructorDescription
StringTokenizer(String str)creates StringTokenizer with specified string.
StringTokenizer(String str, String delim)creates StringTokenizer with specified string and delimeter.
StringTokenizer(String str, String delim, boolean returnValue)creates StringTokenizer with specified string, delimeter and returnValue. If return value is true, delimiter characters are considered to be tokens. If it is false, delimiter characters serve to separate tokens.
Methods of StringTokenizer class

The 6 useful methods of StringTokenizer class are as follows

Public methodDescription
boolean hasMoreTokens()checks if there is more tokens available.
String nextToken()returns the next token from the StringTokenizer object.
String nextToken(String delim)returns the next token based on the delimeter.
boolean hasMoreElements() same as hasMoreTokens() method.
Object nextElement() same as nextToken() but its return type is Object.
int countTokens()returns the total number of tokens.
Simple example of StringTokenizer class

See the simple example of StringTokenizer class that tokenizes a string "my name is khan" on the basis of whitespace.


import java.util.StringTokenizer;  
public class Test1
{  
 public static void main(String args[])
   {  
   StringTokenizer st = new StringTokenizer("my name is Rama"," ");  
     while (st.hasMoreTokens()) 
     {  
         System.out.println(st.nextToken());  
     }  
   }  
}


Output:

my
name
is
Rama
Example of nextToken(String delim) method of StringTokenizer class

import java.util.*;  
  
public class Test1 
{  
   public static void main(String[] args) 
  {  
       StringTokenizer st = new StringTokenizer("my,name,is,Rama");  
        
      // printing next token  
      System.out.println("Next token is : " + st.nextToken(","));  
   }      
}


Output:

Next token is : my



Instagram


Java Programming