Compartir a través de

TokenizerExtensions.ToTokenCounter(Tokenizer, Int32) Method

  • Reference

Definition

Namespace:
Microsoft.Extensions.AI.Evaluation
Assembly:
Microsoft.Extensions.AI.Evaluation.dll
Package:
Microsoft.Extensions.AI.Evaluation v0.9.56-preview

Returns an IEvaluationTokenCounter given the Tokenizer and the inputTokenLimit for a particular AI model / deployment.

public:
[System::Runtime::CompilerServices::Extension]
 static Microsoft::Extensions::AI::Evaluation::IEvaluationTokenCounter ^ ToTokenCounter(Microsoft::ML::Tokenizers::Tokenizer ^ tokenizer, int inputTokenLimit);
public static Microsoft.Extensions.AI.Evaluation.IEvaluationTokenCounter ToTokenCounter (this Microsoft.ML.Tokenizers.Tokenizer tokenizer, int inputTokenLimit);
static member ToTokenCounter : Microsoft.ML.Tokenizers.Tokenizer * int -> Microsoft.Extensions.AI.Evaluation.IEvaluationTokenCounter
<Extension()>
Public Function ToTokenCounter (tokenizer As Tokenizer, inputTokenLimit As Integer) As IEvaluationTokenCounter

Parameters

tokenizer
Tokenizer

The Tokenizer for a particular AI model.

inputTokenLimit
Int32

The threshold of maximum allowed input tokens for a particular AI model / deployment.

Returns

IEvaluationTokenCounter

An IEvaluationTokenCounter for a particular AI model / deployment.

Applies to