We read every piece of feedback, and take your input very seriously.
To see all available qualifiers, see our documentation.
There was an error while loading. Please reload this page.
1 parent a0dfc2d commit a2e3797Copy full SHA for a2e3797
src/transformers/tokenization_utils_base.py
@@ -3054,11 +3054,12 @@ def _pad(
3054
3055
def convert_tokens_to_string(self, tokens: List[str]) -> str:
3056
"""
3057
- Converts a sequence of tokens in a single string. The most simple way to do it is ``" ".join(tokens)`` but
3058
- we often want to remove sub-word tokenization artifacts at the same time.
+ Converts a sequence of tokens in a single string. The most simple way to do it is ``" ".join(tokens)`` but we
+ often want to remove sub-word tokenization artifacts at the same time.
3059
3060
Args:
3061
tokens (:obj:`List[str]`): The token to join in a string.
3062
+
3063
Returns:
3064
:obj:`str`: The joined tokens.
3065
0 commit comments