L2 - Lexical Analysis PDF
L2 - Lexical Analysis PDF
L2 - Lexical Analysis PDF
Shashank Gupta
BITS Pilani Assistant Professor
Pilani Campus
Department of Computer Science and Information Systems
What is Lexical Analysis (LA)?
3
CS F363 Compiler Construction BITS Pilani, Pilani Campus
Tokens, Patterns and Lexemes
5
CS F363 Compiler Construction BITS Pilani, Pilani Campus
Problems in Lexical Analysis
6
CS F363 Compiler Construction BITS Pilani, Pilani Campus
Specification and Recognition of
Tokens
7
CS F363 Compiler Construction BITS Pilani, Pilani Campus
Interface to other Phases
#include <stdio.h>
#include <ctype.h>
//int lineno = 1;
int tokenval = NONE;
int lex() {
int t;
while (1) {
t = getchar ();
if (t = = ' ' || t = = '\t');
else if (t = = '\n')lineno = lineno + 1;
else if (isdigit (t) ) {
tokenval = t - '0' ;
t = getchar ();
while (isdigit(t)) {
tokenval = tokenval * 10 + t - '0' ;
t = getchar();
}
ungetc(t,stdin);
return num;
}
else { tokenval = NONE; return t; }
}
} 10
CS F363 Compiler Construction BITS Pilani, Pilani Campus
Problems
12
CS F363 Compiler Construction BITS Pilani, Pilani Campus
Implementation of Symbol Table
14
CS F363 Compiler Construction BITS Pilani, Pilani Campus
Handling of Keywords
Handling of Blanks
• In Pascal, blanks separate identifiers.
• In Fortran, blanks are important in some
situations.
CS F363 Compiler Construction 16
BITS Pilani, Pilani Campus
PL/1 Problems
Digit 0 |1 | 2 | | 9
Digits Digit
Fraction '.' Digits|
Exponent ( E ( | |) Digits) |
Number Digits Fraction Exponent
All states has an action attached to it, which is executed when the
state is reached. Usually, such actions returns a token and its
attribute value.
CS F363 Compiler Construction 30
BITS Pilani, Pilani Campus
Transition Diagram for Identifier
and Keywords
33
CS F363 Compiler Construction BITS Pilani, Pilani Campus
Transition Diagram for Unsigned
Numbers in C
36
CS F363 Compiler Construction BITS Pilani, Pilani Campus
Example
38
CS F363 Compiler Construction BITS Pilani, Pilani Campus
Transition Diagram for Identifiers
and Keywords
41
CS F363 Compiler Construction BITS Pilani, Pilani Campus
Lexical Analyzer Implementation
from Transition Diagrams
46
CS F363 Compiler Construction BITS Pilani, Pilani Campus
Generation of Lexical Analyzer
from Transition Diagram
• Program that reads input character stream and breaks that into
tokens
• Reports lexical errors (unexpected characters), if any
CS F363 Compiler Construction 48
BITS Pilani, Pilani Campus
Thank You
13
CS F363 Compiler Construction BITS Pilani, Pilani Campus