Skip to content

Commit 6020260

Browse files
lysnikolaoupicnixzAA-TurnerhugovkWingysam
authored
gh-132661: Implement PEP 750 (#132662)
Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com> Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com> Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com> Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com> Co-authored-by: Wingy <git@wingysam.xyz> Co-authored-by: Koudai Aono <koxudaxi@gmail.com> Co-authored-by: Dave Peck <davepeck@gmail.com> Co-authored-by: Terry Jan Reedy <tjreedy@udel.edu> Co-authored-by: Paul Everitt <pauleveritt@me.com> Co-authored-by: sobolevn <mail@sobolevn.me>
1 parent 5ea9010 commit 6020260

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

81 files changed

+6148
-2193
lines changed

‎.github/CODEOWNERS

+6
Original file line numberDiff line numberDiff line change
@@ -320,3 +320,9 @@ Lib/test/test__colorize.py @hugovk
320320

321321
# Fuzzing
322322
Modules/_xxtestfuzz/ @ammaraskar
323+
324+
# t-strings
325+
**/*interpolationobject* @lysnikolaou
326+
**/*templateobject* @lysnikolaou
327+
**/*templatelib* @lysnikolaou
328+
**/*tstring* @lysnikolaou

‎Doc/library/token.rst

+35
Original file line numberDiff line numberDiff line change
@@ -131,6 +131,41 @@ The token constants are:
131131

132132
The token string contains the closing quote(s).
133133

134+
.. data:: TSTRING_START
135+
136+
Token value used to indicate the beginning of a template string literal.
137+
138+
.. impl-detail::
139+
140+
The token string includes the prefix and the opening quote(s), but none
141+
of the contents of the literal.
142+
143+
.. versionadded:: next
144+
145+
.. data:: TSTRING_MIDDLE
146+
147+
Token value used for literal text inside a template string literal
148+
including format specifications.
149+
150+
.. impl-detail::
151+
152+
Replacement fields (that is, the non-literal parts of t-strings) use
153+
the same tokens as other expressions, and are delimited by
154+
:data:`LBRACE`, :data:`RBRACE`, :data:`EXCLAMATION` and :data:`COLON`
155+
tokens.
156+
157+
.. versionadded:: next
158+
159+
.. data:: TSTRING_END
160+
161+
Token value used to indicate the end of a template string literal.
162+
163+
.. impl-detail::
164+
165+
The token string contains the closing quote(s).
166+
167+
.. versionadded:: next
168+
134169
.. data:: ENDMARKER
135170

136171
Token value that indicates the end of input.

‎Doc/whatsnew/3.14.rst

+71
Original file line numberDiff line numberDiff line change
@@ -66,6 +66,7 @@ Summary -- release highlights
6666
6767
* :ref:`PEP 649: deferred evaluation of annotations <whatsnew314-pep649>`
6868
* :ref:`PEP 741: Python Configuration C API <whatsnew314-pep741>`
69+
* :ref:`PEP 750: Template Strings <whatsnew314-pep750>`
6970
* :ref:`PEP 758: Allow except and except* expressions without parentheses <whatsnew314-pep758>`
7071
* :ref:`PEP 761: Discontinuation of PGP signatures <whatsnew314-pep761>`
7172
* :ref:`PEP 765: Disallow return/break/continue that exit a finally block <whatsnew314-pep765>`
@@ -92,6 +93,76 @@ If you encounter :exc:`NameError`\s or pickling errors coming out of
9293
New features
9394
============
9495

96+
.. _whatsnew314-pep750:
97+
98+
PEP 750: Template Strings
99+
-------------------------
100+
101+
Template string literals (t-strings) are a generalization of f-strings,
102+
using a ``t`` in place of the ``f`` prefix. Instead of evaluating
103+
to :class:`str`, t-strings evaluate to a new :class:`!string.templatelib.Template` type:
104+
105+
.. code-block:: python
106+
107+
from string.templatelib import Template
108+
109+
name = "World"
110+
template: Template = t"Hello {name}"
111+
112+
The template can then be combined with functions that operate on the template's
113+
structure to produce a :class:`str` or a string-like result.
114+
For example, sanitizing input:
115+
116+
.. code-block:: python
117+
118+
evil = "<script>alert('evil')</script>"
119+
template = t"<p>{evil}</p>"
120+
assert html(template) == "<p>&lt;script&gt;alert('evil')&lt;/script&gt;</p>"
121+
122+
As another example, generating HTML attributes from data:
123+
124+
.. code-block:: python
125+
126+
attributes = {"src": "shrubbery.jpg", "alt": "looks nice"}
127+
template = t"<img {attributes} />"
128+
assert html(template) == '<img src="shrubbery.jpg" alt="looks nice" class="looks-nice" />'
129+
130+
Unlike f-strings, the ``html`` function has access to template attributes
131+
containing the original information: static strings, interpolations, and values
132+
from the original scope. Unlike existing templating approaches, t-strings build
133+
from the well-known f-string syntax and rules. Template systems thus benefit
134+
from Python tooling as they are much closer to the Python language, syntax,
135+
scoping, and more.
136+
137+
Writing template handlers is straightforward:
138+
139+
.. code-block:: python
140+
141+
from string.templatelib import Template, Interpolation
142+
143+
def lower_upper(template: Template) -> str:
144+
"""Render static parts lowercased and interpolations uppercased."""
145+
parts: list[str] = []
146+
for item in template:
147+
if isinstance(item, Interpolation):
148+
parts.append(str(item.value).upper())
149+
else:
150+
parts.append(item.lower())
151+
return "".join(parts)
152+
153+
name = "world"
154+
assert lower_upper(t"HELLO {name}") == "hello WORLD"
155+
156+
With this in place, developers can write template systems to sanitize SQL, make
157+
safe shell operations, improve logging, tackle modern ideas in web development
158+
(HTML, CSS, and so on), and implement lightweight, custom business DSLs.
159+
160+
See :pep:`750` for more details.
161+
162+
(Contributed by Jim Baker, Guido van Rossum, Paul Everitt, Koudai Aono,
163+
Lysandros Nikolaou, Dave Peck, Adam Turner, Jelle Zijlstra, Bénédikt Tran,
164+
and Pablo Galindo Salgado in :gh:`132661`.)
165+
95166
.. _whatsnew314-pep768:
96167

97168
PEP 768: Safe external debugger interface for CPython

‎Grammar/Tokens

+3
Original file line numberDiff line numberDiff line change
@@ -62,6 +62,9 @@ SOFT_KEYWORD
6262
FSTRING_START
6363
FSTRING_MIDDLE
6464
FSTRING_END
65+
TSTRING_START
66+
TSTRING_MIDDLE
67+
TSTRING_END
6568
COMMENT
6669
NL
6770
ERRORTOKEN

‎Grammar/python.gram

+51-8
Original file line numberDiff line numberDiff line change
@@ -519,7 +519,7 @@ literal_pattern[pattern_ty]:
519519
literal_expr[expr_ty]:
520520
| signed_number !('+' | '-')
521521
| complex_number
522-
| strings
522+
| &(STRING|FSTRING_START|TSTRING_START) strings
523523
| 'None' { _PyAST_Constant(Py_None, NULL, EXTRA) }
524524
| 'True' { _PyAST_Constant(Py_True, NULL, EXTRA) }
525525
| 'False' { _PyAST_Constant(Py_False, NULL, EXTRA) }
@@ -859,7 +859,7 @@ atom[expr_ty]:
859859
| 'True' { _PyAST_Constant(Py_True, NULL, EXTRA) }
860860
| 'False' { _PyAST_Constant(Py_False, NULL, EXTRA) }
861861
| 'None' { _PyAST_Constant(Py_None, NULL, EXTRA) }
862-
| &(STRING|FSTRING_START) strings
862+
| &(STRING|FSTRING_START|TSTRING_START) strings
863863
| NUMBER
864864
| &'(' (tuple | group | genexp)
865865
| &'[' (list | listcomp)
@@ -935,7 +935,7 @@ fstring_middle[expr_ty]:
935935
fstring_replacement_field[expr_ty]:
936936
| '{' a=annotated_rhs debug_expr='='? conversion=[fstring_conversion] format=[fstring_full_format_spec] rbrace='}' {
937937
_PyPegen_formatted_value(p, a, debug_expr, conversion, format, rbrace, EXTRA) }
938-
| invalid_replacement_field
938+
| invalid_fstring_replacement_field
939939
fstring_conversion[ResultTokenWithMetadata*]:
940940
| conv_token="!" conv=NAME { _PyPegen_check_fstring_conversion(p, conv_token, conv) }
941941
fstring_full_format_spec[ResultTokenWithMetadata*]:
@@ -946,8 +946,27 @@ fstring_format_spec[expr_ty]:
946946
fstring[expr_ty]:
947947
| a=FSTRING_START b=fstring_middle* c=FSTRING_END { _PyPegen_joined_str(p, a, (asdl_expr_seq*)b, c) }
948948

949+
tstring_format_spec_replacement_field[expr_ty]:
950+
| '{' a=annotated_rhs debug_expr='='? conversion=[fstring_conversion] format=[tstring_full_format_spec] rbrace='}' {
951+
_PyPegen_formatted_value(p, a, debug_expr, conversion, format, rbrace, EXTRA) }
952+
| invalid_tstring_replacement_field
953+
tstring_format_spec[expr_ty]:
954+
| t=TSTRING_MIDDLE { _PyPegen_decoded_constant_from_token(p, t) }
955+
| tstring_format_spec_replacement_field
956+
tstring_full_format_spec[ResultTokenWithMetadata*]:
957+
| colon=':' spec=tstring_format_spec* { _PyPegen_setup_full_format_spec(p, colon, (asdl_expr_seq *) spec, EXTRA) }
958+
tstring_replacement_field[expr_ty]:
959+
| '{' a=annotated_rhs debug_expr='='? conversion=[fstring_conversion] format=[tstring_full_format_spec] rbrace='}' {
960+
_PyPegen_interpolation(p, a, debug_expr, conversion, format, rbrace, EXTRA) }
961+
| invalid_tstring_replacement_field
962+
tstring_middle[expr_ty]:
963+
| tstring_replacement_field
964+
| t=TSTRING_MIDDLE { _PyPegen_constant_from_token(p, t) }
965+
tstring[expr_ty] (memo):
966+
| a=TSTRING_START b=tstring_middle* c=TSTRING_END { _PyPegen_template_str(p, a, (asdl_expr_seq*)b, c) }
967+
949968
string[expr_ty]: s[Token*]=STRING { _PyPegen_constant_from_string(p, s) }
950-
strings[expr_ty] (memo): a[asdl_expr_seq*]=(fstring|string)+ { _PyPegen_concatenate_strings(p, a, EXTRA) }
969+
strings[expr_ty] (memo): a[asdl_expr_seq*]=(fstring|string|tstring)+ { _PyPegen_concatenate_strings(p, a, EXTRA) }
951970

952971
list[expr_ty]:
953972
| '[' a=[star_named_expressions] ']' { _PyAST_List(a, Load, EXTRA) }
@@ -1212,6 +1231,8 @@ invalid_expression:
12121231
RAISE_SYNTAX_ERROR_KNOWN_LOCATION (a, "expected expression before 'if', but statement is given") }
12131232
| a='lambda' [lambda_params] b=':' &FSTRING_MIDDLE {
12141233
RAISE_SYNTAX_ERROR_KNOWN_RANGE(a, b, "f-string: lambda expressions are not allowed without parentheses") }
1234+
| a='lambda' [lambda_params] b=':' &TSTRING_MIDDLE {
1235+
RAISE_SYNTAX_ERROR_KNOWN_RANGE(a, b, "t-string: lambda expressions are not allowed without parentheses") }
12151236

12161237
invalid_named_expression(memo):
12171238
| a=expression ':=' expression {
@@ -1454,28 +1475,50 @@ invalid_starred_expression_unpacking:
14541475
invalid_starred_expression:
14551476
| '*' { RAISE_SYNTAX_ERROR("Invalid star expression") }
14561477

1457-
invalid_replacement_field:
1478+
invalid_fstring_replacement_field:
14581479
| '{' a='=' { RAISE_SYNTAX_ERROR_KNOWN_LOCATION(a, "f-string: valid expression required before '='") }
14591480
| '{' a='!' { RAISE_SYNTAX_ERROR_KNOWN_LOCATION(a, "f-string: valid expression required before '!'") }
14601481
| '{' a=':' { RAISE_SYNTAX_ERROR_KNOWN_LOCATION(a, "f-string: valid expression required before ':'") }
14611482
| '{' a='}' { RAISE_SYNTAX_ERROR_KNOWN_LOCATION(a, "f-string: valid expression required before '}'") }
1462-
| '{' !annotated_rhs { RAISE_SYNTAX_ERROR_ON_NEXT_TOKEN("f-string: expecting a valid expression after '{'")}
1483+
| '{' !annotated_rhs { RAISE_SYNTAX_ERROR_ON_NEXT_TOKEN("f-string: expecting a valid expression after '{'") }
14631484
| '{' annotated_rhs !('=' | '!' | ':' | '}') {
14641485
PyErr_Occurred() ? NULL : RAISE_SYNTAX_ERROR_ON_NEXT_TOKEN("f-string: expecting '=', or '!', or ':', or '}'") }
14651486
| '{' annotated_rhs '=' !('!' | ':' | '}') {
14661487
PyErr_Occurred() ? NULL : RAISE_SYNTAX_ERROR_ON_NEXT_TOKEN("f-string: expecting '!', or ':', or '}'") }
1467-
| '{' annotated_rhs '='? invalid_conversion_character
1488+
| '{' annotated_rhs '='? invalid_fstring_conversion_character
14681489
| '{' annotated_rhs '='? ['!' NAME] !(':' | '}') {
14691490
PyErr_Occurred() ? NULL : RAISE_SYNTAX_ERROR_ON_NEXT_TOKEN("f-string: expecting ':' or '}'") }
14701491
| '{' annotated_rhs '='? ['!' NAME] ':' fstring_format_spec* !'}' {
14711492
PyErr_Occurred() ? NULL : RAISE_SYNTAX_ERROR_ON_NEXT_TOKEN("f-string: expecting '}', or format specs") }
14721493
| '{' annotated_rhs '='? ['!' NAME] !'}' {
14731494
PyErr_Occurred() ? NULL : RAISE_SYNTAX_ERROR_ON_NEXT_TOKEN("f-string: expecting '}'") }
14741495

1475-
invalid_conversion_character:
1496+
invalid_fstring_conversion_character:
14761497
| '!' &(':' | '}') { RAISE_SYNTAX_ERROR_ON_NEXT_TOKEN("f-string: missing conversion character") }
14771498
| '!' !NAME { RAISE_SYNTAX_ERROR_ON_NEXT_TOKEN("f-string: invalid conversion character") }
14781499

1500+
invalid_tstring_replacement_field:
1501+
| '{' a='=' { RAISE_SYNTAX_ERROR_KNOWN_LOCATION(a, "t-string: valid expression required before '='") }
1502+
| '{' a='!' { RAISE_SYNTAX_ERROR_KNOWN_LOCATION(a, "t-string: valid expression required before '!'") }
1503+
| '{' a=':' { RAISE_SYNTAX_ERROR_KNOWN_LOCATION(a, "t-string: valid expression required before ':'") }
1504+
| '{' a='}' { RAISE_SYNTAX_ERROR_KNOWN_LOCATION(a, "t-string: valid expression required before '}'") }
1505+
| '{' !annotated_rhs { RAISE_SYNTAX_ERROR_ON_NEXT_TOKEN("t-string: expecting a valid expression after '{'") }
1506+
| '{' annotated_rhs !('=' | '!' | ':' | '}') {
1507+
PyErr_Occurred() ? NULL : RAISE_SYNTAX_ERROR_ON_NEXT_TOKEN("t-string: expecting '=', or '!', or ':', or '}'") }
1508+
| '{' annotated_rhs '=' !('!' | ':' | '}') {
1509+
PyErr_Occurred() ? NULL : RAISE_SYNTAX_ERROR_ON_NEXT_TOKEN("t-string: expecting '!', or ':', or '}'") }
1510+
| '{' annotated_rhs '='? invalid_tstring_conversion_character
1511+
| '{' annotated_rhs '='? ['!' NAME] !(':' | '}') {
1512+
PyErr_Occurred() ? NULL : RAISE_SYNTAX_ERROR_ON_NEXT_TOKEN("t-string: expecting ':' or '}'") }
1513+
| '{' annotated_rhs '='? ['!' NAME] ':' fstring_format_spec* !'}' {
1514+
PyErr_Occurred() ? NULL : RAISE_SYNTAX_ERROR_ON_NEXT_TOKEN("t-string: expecting '}', or format specs") }
1515+
| '{' annotated_rhs '='? ['!' NAME] !'}' {
1516+
PyErr_Occurred() ? NULL : RAISE_SYNTAX_ERROR_ON_NEXT_TOKEN("t-string: expecting '}'") }
1517+
1518+
invalid_tstring_conversion_character:
1519+
| '!' &(':' | '}') { RAISE_SYNTAX_ERROR_ON_NEXT_TOKEN("t-string: missing conversion character") }
1520+
| '!' !NAME { RAISE_SYNTAX_ERROR_ON_NEXT_TOKEN("t-string: invalid conversion character") }
1521+
14791522
invalid_arithmetic:
14801523
| sum ('+'|'-'|'*'|'/'|'%'|'//'|'@') a='not' b=inversion { RAISE_SYNTAX_ERROR_KNOWN_RANGE(a, b, "'not' after an operator must be parenthesized") }
14811524
invalid_factor:

‎Include/internal/pycore_ast.h

+21-3
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎Include/internal/pycore_ast_state.h

+3
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎Include/internal/pycore_global_objects_fini_generated.h

+2
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎Include/internal/pycore_global_strings.h

+2
Original file line numberDiff line numberDiff line change
@@ -371,6 +371,7 @@ struct _Py_global_strings {
371371
STRUCT_FOR_ID(consts)
372372
STRUCT_FOR_ID(context)
373373
STRUCT_FOR_ID(contravariant)
374+
STRUCT_FOR_ID(conversion)
374375
STRUCT_FOR_ID(cookie)
375376
STRUCT_FOR_ID(copy)
376377
STRUCT_FOR_ID(copyreg)
@@ -428,6 +429,7 @@ struct _Py_global_strings {
428429
STRUCT_FOR_ID(exception)
429430
STRUCT_FOR_ID(existing_file_name)
430431
STRUCT_FOR_ID(exp)
432+
STRUCT_FOR_ID(expression)
431433
STRUCT_FOR_ID(extend)
432434
STRUCT_FOR_ID(extra_tokens)
433435
STRUCT_FOR_ID(facility)

0 commit comments

Comments
 (0)