Fix None in add_token_positions - issue huggingface#10210 (huggingface#10374)

andreabac3 · web-flow · commit b040e6efc19f · 2021-02-25T09:18:33.000-07:00
* Fix None in add_token_positions - issue huggingface#10210 Fix None in add_token_positions related to the issue huggingface#10210 * add_token_positions fix None values in end_positions vector add_token_positions fix None in end_positions vector as proposed by @joeddav
diff --git a/docs/source/custom_datasets.rst b/docs/source/custom_datasets.rst
@@ -563,6 +563,7 @@ we can use the built in :func:`~transformers.BatchEncoding.char_to_token` method
             # if start position is None, the answer passage has been truncated
             if start_positions[-1] is None:
                 start_positions[-1] = tokenizer.model_max_length
+            if end_positions[-1] is None:
                 end_positions[-1] = tokenizer.model_max_length
 
         encodings.update({'start_positions': start_positions, 'end_positions': end_positions})
