# What are Tokens in programming

## What are Tokens in programming

Smallest individual element of a program is called as Token. Everything you see inside a program is a token.

For example - Suppose an English sentence. "C language is an awesome language. C was developed by Dennis Ritchie at AT&T Bell labs in 1972."

The above sentence is made of Alphabets (a-z A-Z), Blank spaces, Digits (0-9) and special characters (full stop in our case). These are building blocks or basic elements of our sentence. Similarly there are various basic programming elements that makes any program.

## There are five types of tokens.

1.Keyword

2.Identifier

3.Operator

4.Separator

5.Literal

## What are Keywords?

Keyword is a reserved word whose meaning is already defined by the programming language. We cannot use keyword for any other purpose inside programming. Every programming language have some set of keywords. Examples: int, do, while, void, return etc(Note: These keywords are common to C and C influenced languages).

## What are Identifiers?

Identifiers are the name given to different programming elements. Either name given to a variable or a function or any other programming element, all follow some basic naming conventions listed below:

1.Keywords must not be used as an identifier.

2.Identifier must begin with an alphabet a-z A-Z or an underscore_ symbol.

3.Identifier can contains alphabets a-z A-Z, digits 0-9 and underscore _ symbol.

4.Identifier must not contain any special character (e.g. !@$*.'[] etc.) except underscore _.

## Examples of some valid identifiers

num, Num, _num, _Num, num1, Num1, _num1, _Num1, _1num, _1Num, _num_, number_to_add

##### etc.

## Examples of some invalid identifiers

1num, number to add, 1_num, num-to-add, num@

##### etc.

## What are Operator?

Operators are the symbol given to any

arithmetical or logical operations. Various programming languages provides various sets of operators some common operators are:Lets suppose two variables a=10, b=5

- operator subtracts second operand from first i.e. a - b and results 5.

Operator | Description | Example |
---|---|---|

Arithmetic operator | ||

Arithmetic operator are used to perform basic arithmetic operations. | ||

+ | Adds two operand. | a + b gives 15 |

* | Multiplies two operands. | a * b gives 50 |

/ | Divides two operands. | a / b gives 2 |

% | Modulus operator divides the first operand from second and returns the remainder. It is generally used for checking divisibility. | a % b gives 0 (As 10/5 will have 0 remainder) |

## Assignment operator

Assignment operator is used to assign value to a variable. The value is assigned from right to left.

= | Assigns value from right operand to left operand. | a = 10 will assign 10 in a |

## Relational operator

Relational operator are used to check relation between any two operands. Whether any of them is greater, equal or not equal.

> | If value of left operand is greater than right, returns true else returns false | (a > b) will return true |
---|---|---|

< | If value of right operand is greater than left, returns true else returns false | (a < b) will return false |

== | If both operands are equal returns true else false | (a == b) will return false |

!= | If both operands are not equal returns true else false. | (a != b) will return true |

>= | If value of left operand is greater or equal to right operand, returns true else false | (a >= b) will return true |

<= | If value of right operand is greater or equal to left operand, returns true else false | (a <= b) will return false |

## Logical operator

Logical operator are used to combine two boolean expression together and results in a single boolean value according to the operand and operator used.

&& | Used to combine two expressions. If both operands are true or Non-Zero, returns true else false | ((a>=1) && (a<=10)) will return true since (a>=1) is true and also (a<=10) is true. |
---|---|---|

|| | If any of the operand is true or Non-zero, returns true else false | ((a>1) || (a<5)) will return true. As (a>1) is true. Since first operand is true hence there is no need to check for second operand. |

! | Logical NOT operator is a unary operator. Returns the complement of the boolean value. | !(a>1) will return false. Since (a>1) is true hence its complement is false. |

## Bitwise operator

Bitwise operator performs operations on Bits(Binary level). Lets suppose a = 10, b = 5

a = 0000 1010 (8-bit binary representation of 10)

b = 0000 0101 (8-bit binary representation of 5)

& | Bitwise AND performs anding operation on two binary bits value. If both the values are 1 then will result is 1 else will result in 0. | 0000 1010 & 0000 0101 ____________ 0000 0000 |
---|---|---|

| | Bitwise OR returns 1 if any of the two binary bits are 1 else returns 0. | 0000 1010 | 0000 0101 ___________ 0000 1111 |

^ | Bitwise XOR returns 1 if both the binary bits are different else returns 0. | 0000 1010 ^ 0000 0101 ___________ 0000 1111 |

~ | Bitwise COMPLEMENT is a unary operator.It returns the complement of the binary value i.e. if the binary bit is 0 returns 1 else returns 0. | ~ 0000 1010 ___________ 1111 0101 |

<< | Bitwise LEFT SHIFT operator is also unary operator. It shift the binary bits to the left. It inserts a 0 bit value to the extreme right of the binary value. Or we may say it generally multiplies the value with 2. | 0000 1010 << 2 = 0010 1000 |

>> | Bitwise RIGHT SHIFT operator is an unary operator. It shifts the binary bits to the right. It inserts a 0 bit value to the extreme left of the binary value. Or we may say it generally divides the value with 2. | 0000 1010 << 2 = 0000 0010 |

## Increment/Decrement operator

Increment/Decrement operator is a unary operator used to increase an integer value by 1 or decrease it by 1. Increment/decrement operator are of two types Postfix and Prefix.

++ | Increment operator will add 1 to an integer value. | a++ will give 11 ++a will also give 11 |
---|---|---|

-- | Decrement operator will subtract 1 from an integer value. | a-- will give 9 --a will also give 9 |

## Conditional/Ternary operator

Ternary operator as a conditional operator and is similar to simple if-else. It takes three operand.

?: |
It is used as conditional operator. Syntax of using ternary operator: (condition) ? (true part) : (false part) |
b = (a>1) ? a : b; will store the value 10 in b as (a>1) is true hence true part will execute, assigning the value of a in b. |

## What are Separators?

Separators are used to separate different programming elements. The various types of separators used in programming are:

(Space) \t(Tab) \n(New line) . , ; () {} []

## What are Literals?

Literals are constant values that are used for performing various operations and calculations. There are basically three types of literals:

##### 1.Integer literal

An integer literal represents integer or numeric values.

Example: 1, 100, -12312 etc

##### 2.Floating point literal

Floating point literal represents fractional values.

Example: 2.123, 1.02, -2.33, 13e54, -23.3 etc

##### 3.Character literal

Character literal represent character values. Single character are enclosed in a single quote(' ') while sequence of character are enclosed in double quotes(" ")

Example: 'a', 'n', "Hello", "Hello123" etc.