Tokenizer.DefaultTokenizerImpl(String, String) Method
Definition
Important
Some information relates to prerelease product that may be substantially modified before it’s released. Microsoft makes no warranties, express or implied, with respect to the information provided here.
Simple tokenizer that breaks on spaces and punctuation. The only normalization done is to lowercase.
public static System.Collections.Generic.List<Microsoft.Bot.Builder.Dialogs.Choices.Token> DefaultTokenizerImpl (string text, string locale = default);static member DefaultTokenizerImpl : string * string -> System.Collections.Generic.List<Microsoft.Bot.Builder.Dialogs.Choices.Token>Public Shared Function DefaultTokenizerImpl (text As String, Optional locale As String = Nothing) As List(Of Token)Parameters
- text
- String
The input text.
- locale
- String
Optional, identifies the locale of the input text.
Returns
A list of tokens.
Remarks
This is an exact port of the JavaScript implementation of the algorithm except that here the .NET library functions are used in place of the JavaScript string code point functions.