SrcForge

Lesson 8

Tokens

Covers keywords, identifiers, literals, operators, punctuators.

Lesson content

What are tokens?

Token is smallest unit compiler understands.

Program is built from tokens, like sentences from words.

int sum = 10 + 20;
Token | Type
----- | ----------------------
int   | Keyword
sum   | Identifier
=     | Operator
10    | Literal
+     | Operator
20    | Literal
;     | Separator (Punctuator)

Why are tokens important?

Compiler breaks code into tokens first.

  • Learn C++ syntax more easily
  • Write error-free programs
  • Understand compiler error messages
  • Improve code readability

Classification of C++ tokens

                    C++ Tokens

 ┌─────────────┬─────────────┬─────────────┬─────────────┐
 │             │             │             │
 ▼             ▼             ▼             ▼
Keywords   Identifiers   Literals     Operators


                  Punctuators
                  (Separators)

1. Keywords

Reserved words with fixed meaning. Can't be identifiers.

Keyword | Purpose
------- | ---------------------------
int     | Integer data type
float   | Floating-point data type
char    | Character data type
double  | Double-precision data type
bool    | Boolean data type
if      | Conditional statement
else    | Alternative condition
for     | Loop
while   | Loop
return  | Return from a function
class   | Define a class
const   | Declare a constant
void    | No return value
int age = 20;

2. Identifiers

User-defined names for vars, functions, classes.

int studentAge = 18;
  • Start with letter or underscore
  • Can contain letters, digits, underscores
  • No spaces allowed
  • No C++ keywords allowed
  • Case-sensitive names
// Valid
age
studentName
_total
marks2025

// Invalid
1student
student name
float
total-score

3. Literals

Fixed value written directly in code.

Literal Type   | Example
-------------- | -------
Integer        | 100
Floating-point | 3.14
Character      | 'A'
String         | "Hello"
Boolean        | true
int age = 25;

4. Operators

Symbols performing operations on operands.

Operator | Description
-------- | --------------------
+        | Addition
-        | Subtraction
*        | Multiplication
/        | Division
%        | Modulus
=        | Assignment
==       | Equality comparison
&&       | Logical AND
||       | Logical OR
int sum = 10 + 20;

5. Punctuators (separators)

Symbols separating code and defining structure.

Symbol | Purpose
------ | -------------------------------
;      | Ends a statement
,      | Separates items
{}     | Defines a block of code
()     | Function declaration and calls
[]     | Array indexing
int main()
{
    return 0;
}

Complete example

#include <iostream>
using namespace std;

int main()
{
    const int age = 20;
    float height = 5.8f;

    cout << "Age: " << age << endl;
    cout << "Height: " << height;

    return 0;
}

Breakdown of tokens

const int age = 20;
Token | Category
----- | ----------------
const | Keyword
int   | Keyword
age   | Identifier
=     | Operator
20    | Integer Literal
;     | Punctuator

Common mistakes and pitfalls

  • Keywords as identifiers. Fix: use valid names
  • Identifier starts with digit. Fix: start with letter
  • Forgetting semicolons. Fix: end statements with ;
  • Double quotes for char. Fix: use single quotes
// Incorrect
int float = 10;
// Correct
int number = 10;

Best practices

  • Use meaningful identifier names
  • Avoid short or unclear names
  • Follow consistent naming convention
  • Never use keywords as identifiers
  • Keep code properly formatted

When not to use certain tokens

  • Keywords as variable names. Use meaningful identifiers
  • Single-char names in big programs. Use descriptive names
  • Magic numbers repeated often. Use named constants

Summary

  • Token is smallest program unit
  • Compiler tokenizes before compiling
  • Five categories: keywords, identifiers, literals, operators, punctuators
  • Tokens help write correct programs
  • Every statement is built from tokens

Frequently asked questions

What is a token in C++?

Smallest unit compiler recognizes during compilation.

What are the different types of tokens in C++?

  • Keywords
  • Identifiers
  • Literals
  • Operators
  • Punctuators (Separators)

What is the difference between a keyword and an identifier?

Keyword is reserved. Identifier is user-defined name.

Is a semicolon a token?

Yes. It's a punctuator marking statement end.

Why should I learn tokens before programming?

Builds foundation for syntax, errors, and OOP concepts.