Lesson 8
Tokens
Covers keywords, identifiers, literals, operators, punctuators.
Lesson content
What are tokens?
Token is smallest unit compiler understands.
Program is built from tokens, like sentences from words.
int sum = 10 + 20;Token | Type
----- | ----------------------
int | Keyword
sum | Identifier
= | Operator
10 | Literal
+ | Operator
20 | Literal
; | Separator (Punctuator)Why are tokens important?
Compiler breaks code into tokens first.
- Learn C++ syntax more easily
- Write error-free programs
- Understand compiler error messages
- Improve code readability
Classification of C++ tokens
C++ Tokens
│
┌─────────────┬─────────────┬─────────────┬─────────────┐
│ │ │ │
▼ ▼ ▼ ▼
Keywords Identifiers Literals Operators
│
▼
Punctuators
(Separators)1. Keywords
Reserved words with fixed meaning. Can't be identifiers.
Keyword | Purpose
------- | ---------------------------
int | Integer data type
float | Floating-point data type
char | Character data type
double | Double-precision data type
bool | Boolean data type
if | Conditional statement
else | Alternative condition
for | Loop
while | Loop
return | Return from a function
class | Define a class
const | Declare a constant
void | No return valueint age = 20;2. Identifiers
User-defined names for vars, functions, classes.
int studentAge = 18;- Start with letter or underscore
- Can contain letters, digits, underscores
- No spaces allowed
- No C++ keywords allowed
- Case-sensitive names
// Valid
age
studentName
_total
marks2025
// Invalid
1student
student name
float
total-score3. Literals
Fixed value written directly in code.
Literal Type | Example
-------------- | -------
Integer | 100
Floating-point | 3.14
Character | 'A'
String | "Hello"
Boolean | trueint age = 25;4. Operators
Symbols performing operations on operands.
Operator | Description
-------- | --------------------
+ | Addition
- | Subtraction
* | Multiplication
/ | Division
% | Modulus
= | Assignment
== | Equality comparison
&& | Logical AND
|| | Logical ORint sum = 10 + 20;5. Punctuators (separators)
Symbols separating code and defining structure.
Symbol | Purpose
------ | -------------------------------
; | Ends a statement
, | Separates items
{} | Defines a block of code
() | Function declaration and calls
[] | Array indexingint main()
{
return 0;
}Complete example
#include <iostream>
using namespace std;
int main()
{
const int age = 20;
float height = 5.8f;
cout << "Age: " << age << endl;
cout << "Height: " << height;
return 0;
}Breakdown of tokens
const int age = 20;Token | Category
----- | ----------------
const | Keyword
int | Keyword
age | Identifier
= | Operator
20 | Integer Literal
; | PunctuatorCommon mistakes and pitfalls
- Keywords as identifiers. Fix: use valid names
- Identifier starts with digit. Fix: start with letter
- Forgetting semicolons. Fix: end statements with ;
- Double quotes for char. Fix: use single quotes
// Incorrect
int float = 10;// Correct
int number = 10;Best practices
- Use meaningful identifier names
- Avoid short or unclear names
- Follow consistent naming convention
- Never use keywords as identifiers
- Keep code properly formatted
When not to use certain tokens
- Keywords as variable names. Use meaningful identifiers
- Single-char names in big programs. Use descriptive names
- Magic numbers repeated often. Use named constants
Summary
- Token is smallest program unit
- Compiler tokenizes before compiling
- Five categories: keywords, identifiers, literals, operators, punctuators
- Tokens help write correct programs
- Every statement is built from tokens
Frequently asked questions
What is a token in C++?
Smallest unit compiler recognizes during compilation.
What are the different types of tokens in C++?
- Keywords
- Identifiers
- Literals
- Operators
- Punctuators (Separators)
What is the difference between a keyword and an identifier?
Keyword is reserved. Identifier is user-defined name.
Is a semicolon a token?
Yes. It's a punctuator marking statement end.
Why should I learn tokens before programming?
Builds foundation for syntax, errors, and OOP concepts.