GPTTokenizer class

Package:

@microsoft/teams-ai

Tokenizer that uses GPT's cl100k_base encoding. To use GPT-3 encoding, pass in an instance of GPT3Tokenizer.

Methods

decode(number[])
encode(string)

Method Details

decode(number[])

function decode(tokens: number[]): string

Parameters

tokens

number[]

Returns

string

encode(string)

function encode(text: string): number[]

Parameters

text

string

Returns

number[]