Skip to content

Conversation

jserv
Copy link
Collaborator

@jserv jserv commented Aug 21, 2025

This commit implements token_pool_alloc()/free() with freelist pattern for memory reuse and adds circular buffer implementation for 8-token lookahead capability. Then, it implements source location tracking via tokenization process and adds enhanced error reporting with file:line:column format.

Summary by Bito

This pull request enhances the lexer by implementing a token pool and circular buffer for improved memory reuse and 8-token lookahead capabilities. It also introduces new structures for source location tracking and enhances error reporting, optimizing the tokenization process and debugging capabilities.

- Add token_info_t structure with type, value, and source location
  tracking
- Add source_location_t for precise error reporting (line, column,
  filename)
- Add token_pool_t for freelist-based memory reuse pattern
- Add token_buffer_t for 8-token circular buffer lookahead capability
This commit implements token_pool_alloc()/free() with freelist pattern
for memory reuse and adds circular buffer implementation for 8-token
lookahead capability. Then, it implements source location tracking via
tokenization process and adds enhanced error reporting with
file:line:column format.

Memory management benefits:
- Each reused token saves ~284 bytes of memory
- Fixed overhead < 2.5KB for buffer and pool structures
- Zero memory leaks through arena-based allocation
@jserv jserv merged commit 48d2b1a into master Aug 24, 2025
12 checks passed
@jserv jserv deleted the token-mm branch August 24, 2025 11:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants