Edited 210_Identifying_words/20_Standard_tokenizer.asciidoc with Atlas code editor

Author: skalapurakkel

210_Identifying_words/20_Standard_tokenizer.asciidoc

@@ -1,13 +1,13 @@
 [[standard-tokenizer]]
-=== standard tokenizer
+=== standard Tokenizer
 
-A tokenizer accepts a string as input, processes((("words", "identifying", "using standard tokenizer")))((("standard tokenizer")))((("tokenizers"))) the string to break it up
-into individual words or _tokens_ (perhaps discarding some characters like
+A _tokenizer_ accepts a string as input, processes((("words", "identifying", "using standard tokenizer")))((("standard tokenizer")))((("tokenizers"))) the string to break it
+into individual words, or _tokens_ (perhaps discarding some characters like
 punctuation), and emits a _token stream_ as output.
 
-What is interesting is the algorithm that is used to *identify* words. The
-`whitespace` tokenizer ((("whitespace tokenizer")))simply breaks on whitespace -- spaces, tabs, line
-feeds etc. -- and assumes that contiguous non-whitespace characters form a
+What is interesting is the algorithm that is used to _identify_ words. The
+`whitespace` tokenizer ((("whitespace tokenizer")))simply breaks on whitespace--spaces, tabs, line
+feeds, and so forth--and assumes that contiguous nonwhitespace characters form a
 single token. For instance:
 
 [source,js]
@@ -16,40 +16,39 @@ GET /_analyze?tokenizer=whitespace
 You're the 1st runner home!
 --------------------------------------------------
 
-The above request would return the following terms:
+This request would return the following terms:
 `You're`, `the`, `1st`, `runner`, `home!`
 
-The `letter` tokenizer, on the other hand, breaks on any character which is
+The `letter` tokenizer, on the other hand, breaks on any character that is
 not a letter, and so would ((("letter tokenizer")))return the following terms: `You`, `re`, `the`,
-`st`, `runner`,`home`
+`st`, `runner`, `home`.
 
 The `standard` tokenizer((("Unicode Text Segmentation algorithm"))) uses the Unicode Text Segmentation algorithm (as
 defined in http://unicode.org/reports/tr29/[Unicode Standard Annex #29]) to
-find the boundaries *between* words,((("word boundaries"))) and emits everything in-between. Its
+find the boundaries _between_ words,((("word boundaries"))) and emits everything in-between. Its
 knowledge of Unicode allows it to successfully tokenize text containing a
 mixture of languages.
 
-Punctuation may((("punctuation", "in words"))) or may not be considered to be part of a word, depending on
+Punctuation may((("punctuation", "in words"))) or may not be considered part of a word, depending on
 where it appears:
 
 [source,js]
 --------------------------------------------------
 GET /_analyze?tokenizer=standard
-You're my 'favourite'.
+You're my 'favorite'.
 --------------------------------------------------
 
-In the above example, the apostrophe in `You're` is treated as part of the
-word while the single quotes in `'favourite'` are not, resulting in the
-following terms: `You're`, `my`, `favourite`.
+In this example, the apostrophe in `You're` is treated as part of the
+word, while the single quotes in `'favorite'` are not, resulting in the
+following terms: `You're`, `my`, `favorite`.
 
 [TIP]
-.`uax_url_email` tokenizer
 ==================================================
 
 The `uax_url_email` tokenizer works((("uax_url_email tokenizer"))) in exactly the same way as the `standard`
-tokenizer, except that it recognises((("email addresses and URLs, tokenizer for"))) email addresses and URLs as emits them as
+tokenizer, except that it recognizes((("email addresses and URLs, tokenizer for"))) email addresses and URLs and emits them as
 single tokens. The `standard` tokenizer, on the other hand, would try to
-break them up into individual words. For instance, the email address
+break them into individual words. For instance, the email address
 `joe-bloggs@foo-bar.com` would result in the tokens `joe`, `bloggs`, `foo`,
 `bar.com`.
 
@@ -58,7 +57,5 @@ break them up into individual words. For instance, the email address
 The `standard` tokenizer is a reasonable starting point for tokenizing most
 languages, especially Western languages.  In fact, it forms the basis of most
 of the language-specific analyzers like the `english`, `french`, and `spanish`
-analyzers.
-
-Its support for Asian languages, however, is limited and you should consider
-using the `icu_tokenizer` instead,((("icu_tokenizer"))) which is available in the ICU plugin.
+analyzers. Its support for Asian languages, however, is limited, and you should consider
+using the `icu_tokenizer` instead,((("icu_tokenizer"))) which is available in the ICU plug-in.