Skip to content

Commit 71922cb

Browse files
committed
Add REPL doc
1 parent e686bfa commit 71922cb

File tree

1 file changed

+32
-0
lines changed
  • lib/node_modules/@stdlib/nlp/tokenize/docs

1 file changed

+32
-0
lines changed
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
2+
{{alias}}( str[, keepWhitespace] )
3+
Tokenizes a string.
4+
5+
To include whitespace characters (spaces, tabs, line breaks) in the output
6+
array, set `keepWhitespace` to `true`.
7+
8+
Parameters
9+
----------
10+
str: string
11+
Input string.
12+
13+
keepWhitespace: boolean
14+
Boolean indicating whether whitespace characters should be returned as
15+
part of the token array. Default: `false`.
16+
17+
Returns
18+
-------
19+
out: Array
20+
Array of tokens.
21+
22+
Examples
23+
--------
24+
> var out = {{alias}}( 'Hello Mrs. Maple, could you call me back?' )
25+
[ 'Hello', 'Mrs.', 'Maple', ',', 'could', 'you', 'call', 'me', 'back', '?' ]
26+
27+
> out = {{alias}}( 'Hello World!', true )
28+
[ 'Hello', ' ', 'World', '!' ]
29+
30+
See Also
31+
--------
32+

0 commit comments

Comments
 (0)