Skip to content

Commit 201db80

Browse files
committed
Remove formatting ?? in odd place.
1 parent b8458ad commit 201db80

File tree

1 file changed

+5
-3
lines changed

1 file changed

+5
-3
lines changed

src/transformers/tokenization_utils_base.py

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -3163,11 +3163,13 @@ def get_special_tokens_mask(
31633163
@staticmethod
31643164
def clean_up_tokenization(out_string: str) -> str:
31653165
"""
3166-
clean up a list of simple english tokenization artifacts like spaces before punctuations and abbreviated forms.
3166+
Clean up a list of simple English tokenization artifacts like spaces before punctuations and abbreviated forms.
31673167
3168-
args: out_string (:obj:`str`): the text to clean up.
3168+
Args:
3169+
out_string (:obj:`str`): the text to clean up.
31693170
3170-
returns: :obj:`str`: the cleaned-up string.
3171+
Returns:
3172+
:obj:`str`: the cleaned-up string.
31713173
"""
31723174
out_string = (
31733175
out_string.replace(" .", ".")

0 commit comments

Comments
 (0)