GPT3Tokenizer class

Tokenizer that uses GPT-3's encoder.

Methods

decode(number[])
encode(string)

Method Details

decode(number[])

function decode(tokens: number[]): string

Parameters

tokens

number[]

Returns

string

encode(string)

function encode(text: string): number[]

Parameters

text

string

Returns

number[]