Docs: Corrected behaviour of max_token_length in standard tokenizer

clintongormley · clintongormley · commit 5fc4908bc6ae · 2016-03-18T10:57:50.000+01:00
diff --git a/docs/reference/analysis/tokenizers/standard-tokenizer.asciidoc b/docs/reference/analysis/tokenizers/standard-tokenizer.asciidoc
@@ -13,6 +13,6 @@ type:
 |=======================================================================
 |Setting |Description
 |`max_token_length` |The maximum token length. If a token is seen that
-exceeds this length then it is discarded. Defaults to `255`.
+exceeds this length then it is split at `max_token_length` intervals. Defaults to `255`.
 |=======================================================================
 
