Tokenizer(Model, PreTokenizer, Normalizer) Constructor
Definition
Important
Some information relates to prerelease product that may be substantially modified before it’s released. Microsoft makes no warranties, express or implied, with respect to the information provided here.
Create a new Tokenizer object.
public Tokenizer (Microsoft.ML.Tokenizers.Model model, Microsoft.ML.Tokenizers.PreTokenizer? preTokenizer = default, Microsoft.ML.Tokenizers.Normalizer? normalizer = default);new Microsoft.ML.Tokenizers.Tokenizer : Microsoft.ML.Tokenizers.Model * Microsoft.ML.Tokenizers.PreTokenizer * Microsoft.ML.Tokenizers.Normalizer -> Microsoft.ML.Tokenizers.TokenizerPublic Sub New (model As Model, Optional preTokenizer As PreTokenizer = Nothing, Optional normalizer As Normalizer = Nothing)Parameters
- model
- Model
The Model in use by the Tokenizer.
- preTokenizer
- PreTokenizer
The optional PreTokenizer in use by the Tokenizer. WhiteSpace PreTokenizer will be used if this parameter is null.
- normalizer
- Normalizer
The optional Normalizer in use by the Tokenizer.