Merge pull request #678 from joshuar/fix_analyze_API_usage

Update Getting Started with Languages section.

Author: joshuar

200_Language_intro/00_Intro.asciidoc

@@ -2,16 +2,10 @@
 == Getting Started with Languages
 
 Elasticsearch ships with a collection of language analyzers that provide
-good, basic, out-of-the-box ((("language analyzers")))((("languages", "getting started with")))support for many of the world's most common
-languages:
+good, basic, https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-lang-analyzer.html[out-of-the-box support]
+for many of the world's most common languages.
 
-Arabic, Armenian, Basque, Brazilian, Bulgarian, Catalan, Chinese,
-Czech, Danish, Dutch, English, Finnish, French, Galician, German, Greek,
-Hindi, Hungarian, Indonesian, Irish, Italian, Japanese, Korean, Kurdish, 
-Norwegian, Persian, Portuguese, Romanian, Russian, Spanish, Swedish, 
-Turkish, and Thai.
-
-These analyzers typically((("language analyzers", "roles performed by"))) perform four roles:
+These analyzers typically perform four roles:
 
 * Tokenize text into individual words:
 +
@@ -30,19 +24,18 @@ These analyzers typically((("language analyzers", "roles performed by"))) perfor
 `foxes` -> `fox`
 
 Each analyzer may also apply other transformations specific to its language in
-order to make words from that((("language analyzers", "other transformations specific to the language"))) language more searchable:
+order to make words from that language more searchable:
 
-* The `english` analyzer ((("english analyzer")))removes the possessive `'s`:
+* The `english` analyzer removes the possessive `'s`:
 +
 `John's` -> `john`
 
-* The `french` analyzer ((("french analyzer")))removes _elisions_ like `l'` and `qu'` and
+* The `french` analyzer removes _elisions_ like `l'` and `qu'` and
   _diacritics_ like `¨` or `^`:
 +
 `l'église` -> `eglis`
 
-* The `german` analyzer normalizes((("german analyzer"))) terms, replacing `ä` and `ae` with `a`, or
+* The `german` analyzer normalizes terms, replacing `ä` and `ae` with `a`, or
   `ß` with `ss`, among others:
 +
 `äußerst` -> `ausserst`
-

200_Language_intro/10_Using.asciidoc

@@ -2,7 +2,7 @@
 === Using Language Analyzers
 
 The built-in language analyzers are available globally and don't need to be
-configured before being used.((("language analyzers", "using")))  They can be specified directly in the field
+configured before being used.  They can be specified directly in the field
 mapping:
 
 [source,js]
@@ -13,26 +13,33 @@ PUT /my_index
     "blog": {
       "properties": {
         "title": {
-          "type":     "string",
+          "type":     "text",
           "analyzer": "english" <1>
         }
       }
     }
   }
 }
 --------------------------------------------------
+// CONSOLE
+
 <1> The `title` field will use the `english` analyzer instead of the default
     `standard` analyzer.
 
-Of course, by passing ((("english analyzer", "information lost with")))text through the `english` analyzer, we lose
-information:
+Of course, by passing text through the `english` analyzer, we lose information:
 
 [source,js]
 --------------------------------------------------
-GET /my_index/_analyze?field=title <1>
-I'm not happy about the foxes
+GET /my_index/_analyze
+{
+  "field": "title"
+  "text": "I'm not happy about the foxes" <1>
+}
 --------------------------------------------------
-<1> Emits token: `i'm`, `happi`, `about`, `fox`
+// CONSOLE
+// TEST[continued]
+
+<1> Emits the tokens: `i'm`, `happi`, `about`, `fox`
 
 We can't tell if the document mentions one `fox` or many  `foxes`; the word
 `not` is a stopword and is removed, so we can't tell whether the document is
@@ -41,7 +48,7 @@ recall as we can match more loosely, but we have reduced our ability to rank
 documents accurately.
 
 To get the best of both worlds, we can use <<multi-fields,multifields>> to
-index the `title` field twice: once((("multifields", "using to index a field with two different analyzers"))) with the `english` analyzer and once with
+index the `title` field twice: once with the `english` analyzer and once with
 the `standard` analyzer:
 
 [source,js]
@@ -52,10 +59,10 @@ PUT /my_index
     "blog": {
       "properties": {
         "title": { <1>
-          "type": "string",
+          "type": "text",
           "fields": {
             "english": { <2>
-              "type":     "string",
+              "type":     "text",
               "analyzer": "english"
             }
           }
@@ -65,6 +72,8 @@ PUT /my_index
   }
 }
 --------------------------------------------------
+// CONSOLE
+
 <1> The main `title` field uses the `standard` analyzer.
 <2> The `title.english` subfield uses the `english` analyzer.
 
@@ -90,12 +99,13 @@ GET /_search
   }
 }
 --------------------------------------------------
+// CONSOLE
+// TEST[continued]
+
 <1> Use the <<most-fields,`most_fields`>> query type to match the
     same text in as many fields as possible.
 
-Even ((("most fields queries")))though neither of our documents contain the word `foxes`,  both documents
-are returned as results thanks to the word stemming on the `title.english`
-field.  The second document is ranked as more relevant, because the word `not`
-matches on the `title` field.
-
-
+Even though neither of our documents contain the
+word `foxes`,  both documents are returned as results thanks to the word
+stemming on the `title.english` field.  The second document is ranked as more
+relevant, because the word `not` matches on the `title` field.

200_Language_intro/20_Configuring.asciidoc

@@ -2,13 +2,13 @@
 === Configuring Language Analyzers
 
 While the language analyzers can be used out of the box without any
-configuration, most of them ((("english analyzer", "configuring")))((("language analyzers", "configuring")))do allow you to control aspects of their
+configuration, most of them do allow you to control aspects of their
 behavior, specifically:
 
 [[stem-exclusion]]
 Stem-word exclusion::
 +
-Imagine, for instance, that users searching for((("language analyzers", "configuring", "stem word exclusion")))((("stemming words", "stem word exclusion, configuring"))) the ``World Health
+Imagine, for instance, that users searching for the ``World Health
 Organization'' are instead getting results for ``organ health.'' The reason
 for this confusion is that both ``organ'' and ``organization'' are stemmed to
 the same root word: `organ`. Often this isn't a problem, but in this
@@ -18,7 +18,7 @@ stemmed.
 
 Custom stopwords::
 
-The default list of stopwords((("stopwords", "configuring for language analyzers"))) used in English are as follows:
+The default list of stopwords used in English are as follows:
 +
     a, an, and, are, as, at, be, but, by, for, if, in, into, is, it,
     no, not, of, on, or, such, that, the, their, then, there, these,
@@ -54,13 +54,17 @@ PUT /my_index
   }
 }
 
-GET /my_index/_analyze?analyzer=my_english <3>
-The World Health Organization does not sell organs.
+GET /my_index/_analyze
+{
+  "analyzer": "my_english", <3>
+  "text": "The World Health Organization does not sell organs."
+}
 --------------------------------------------------
+// CONSOLE
+
 <1> Prevents `organization` and `organizations` from being stemmed
 <2> Specifies a custom list of stopwords
-<3> Emits tokens `world`, `health`, `organization`, `does`, `not`, `sell`, `organ`
+<3> Emits tokens `world`, `health`, `organization`, `doe`, `not`, `sell`, `organ`
 
 We discuss stemming and stopwords in much more detail in <<stemming>> and
 <<stopwords>>, respectively.
-

200_Language_intro/30_Language_pitfalls.asciidoc

@@ -1,9 +1,9 @@
 [[language-pitfalls]]
 === Pitfalls of Mixing Languages
 
-If you have to deal with only a single language,((("languages", "mixing, pitfalls of"))) count yourself lucky.
+If you have to deal with only a single language, count yourself lucky.
 Finding the right strategy for handling documents written in several languages
-can be challenging.((("indexing", "mixed languages, pitfalls of")))
+can be challenging.
 
 ==== At Index Time
 
@@ -21,11 +21,11 @@ separate.  Mixing languages in the same inverted index can be problematic.
 ===== Incorrect stemming
 
 The stemming rules for German are different from those for English, French,
-Swedish, and so on.((("stemming words", "incorrect stemming in multilingual documents"))) Applying the same stemming rules to different languages
+Swedish, and so on. Applying the same stemming rules to different languages
 will result in some words being stemmed correctly, some  incorrectly, and some
-not being stemmed at all. It may even result in words from different languages with different meanings
-being stemmed to the same root word, conflating their meanings and producing
-confusing search results for the user.
+not being stemmed at all. It may even result in words from different languages
+with different meanings being stemmed to the same root word, conflating their
+meanings and producing confusing search results for the user.
 
 Applying multiple stemmers in turn to the same text is likely to result in
 rubbish, as the next stemmer may try to stem an already stemmed word,
@@ -49,7 +49,7 @@ text.
 ===== Incorrect inverse document frequencies
 
 In <<relevance-intro>>, we explained that the more frequently a term appears
-in a collection of documents, the less weight that term has.((("inverse document frequency", "incorrect, in multilingual documents")))  For accurate
+in a collection of documents, the less weight that term has.  For accurate
 relevance calculations, you need accurate term-frequency statistics.
 
 A short snippet of German appearing in predominantly English text would give
@@ -59,11 +59,11 @@ snippets now have much less weight.
 
 ==== At Query Time
 
-It is not sufficient just to think about your documents, though.((("queries", "mixed languages and")))  You also need
-to think about how your users will query those documents.  Often you will be able
-to identify the main language of the user either from the language of that user's chosen
-interface (for example, `mysite.de` versus `mysite.fr`) or from the
-http://www.w3.org/International/questions/qa-lang-priorities.en.php[`accept-language`]
+It is not sufficient just to think about your documents, though.  You also need
+to think about how your users will query those documents.  Often you will be
+able to identify the main language of the user either from the language of that
+user's chosen interface (for example, `mysite.de` versus `mysite.fr`) or from
+the http://www.w3.org/International/questions/qa-lang-priorities.en.php[`accept-language`]
 HTTP header from the user's browser.
 
 User searches also come in three main varieties:
@@ -72,7 +72,8 @@ User searches also come in three main varieties:
 * Users search for words in a different language, but expect results in
   their main language.
 * Users search for words in a different language, and expect results in
-  that language (for example, a bilingual person, or a foreign visitor in a web cafe).
+  that language (for example, a bilingual person, or a foreign visitor in a web
+  cafe).
 
 Depending on the type of data that you are searching, it may be appropriate to
 return results in a single language (for example, a user searching for products on
@@ -102,7 +103,7 @@ library from
 http://blog.mikemccandless.com/2013/08/a-new-version-of-compact-language.html[Mike McCandless],
 which uses the open source (http://www.apache.org/licenses/LICENSE-2.0[Apache License 2.0])
 https://code.google.com/p/cld2/[Compact Language Detector] (CLD) from Google.  It is
-small, fast, ((("Compact Language Detector (CLD)")))and accurate, and can detect 160+ languages from as little as two
+small, fast, and accurate, and can detect 160+ languages from as little as two
 sentences. It can even detect multiple languages within a single block of
 text. Bindings exist for several languages including Python, Perl, JavaScript,
 PHP, C#/.NET, and R.
@@ -113,4 +114,3 @@ Shorter amounts of text, such as search keywords, produce much less accurate
 results. In these cases, it may be preferable to take simple heuristics into
 account such as the country of origin, the user's selected language, and the
 HTTP `accept-language` headers.
-