TextCatalog.ProduceNgrams Méthode

Définition

Espace de noms:

Microsoft.ML

Assembly:

Microsoft.ML.Transforms.dll

Paquet:

Microsoft.ML v3.0.1

Crée un NgramExtractingEstimator vecteur de nombres de n-grammes (séquences de mots consécutifs) rencontré dans le texte d’entrée.

public static Microsoft.ML.Transforms.Text.NgramExtractingEstimator ProduceNgrams (this Microsoft.ML.TransformsCatalog.TextTransforms catalog, string outputColumnName, string inputColumnName = default, int ngramLength = 2, int skipLength = 0, bool useAllLengths = true, int maximumNgramsCount = 10000000, Microsoft.ML.Transforms.Text.NgramExtractingEstimator.WeightingCriteria weighting = Microsoft.ML.Transforms.Text.NgramExtractingEstimator+WeightingCriteria.Tf);

static member ProduceNgrams : Microsoft.ML.TransformsCatalog.TextTransforms * string * string * int * int * bool * int * Microsoft.ML.Transforms.Text.NgramExtractingEstimator.WeightingCriteria -> Microsoft.ML.Transforms.Text.NgramExtractingEstimator

<Extension()>
Public Function ProduceNgrams (catalog As TransformsCatalog.TextTransforms, outputColumnName As String, Optional inputColumnName As String = Nothing, Optional ngramLength As Integer = 2, Optional skipLength As Integer = 0, Optional useAllLengths As Boolean = true, Optional maximumNgramsCount As Integer = 10000000, Optional weighting As NgramExtractingEstimator.WeightingCriteria = Microsoft.ML.Transforms.Text.NgramExtractingEstimator+WeightingCriteria.Tf) As NgramExtractingEstimator

Paramètres

catalog

TransformsCatalog.TextTransforms

Catalogue de transformation liée au texte.

outputColumnName

String

Nom de la colonne résultant de la transformation de inputColumnName. Le type de données de cette colonne sera un vecteur de Single.

inputColumnName

String

Nom de la colonne à transformer. Si elle est définie sur null, la valeur du outputColumnName fichier sera utilisée comme source. Cet estimateur fonctionne sur des vecteurs de type de données clés.

ngramLength

Int32

Longueur de Ngram.

skipLength

Int32

Nombre de jetons à ignorer entre chaque n-gramme. Par défaut, aucun jeton n’est ignoré.

useAllLengths

Boolean

Qu’il s’agisse d’inclure toutes les longueurs n-gram jusqu’à ngramLength ou seulement ngramLength.

maximumNgramsCount

Int32

Nombre maximal de n-grammes à stocker dans le dictionnaire.

weighting

NgramExtractingEstimator.WeightingCriteria

Mesure statistique utilisée pour évaluer l’importance d’un mot ou d’un n-gram dans un document dans un corpus. Lorsqu’elle maximumNgramsCount est inférieure au nombre total de n-grammes rencontrés, cette mesure est utilisée pour déterminer les n-grammes à conserver.

Retours

NgramExtractingEstimator

Exemples

using System;
using System.Collections.Generic;
using Microsoft.ML;
using Microsoft.ML.Data;
using Microsoft.ML.Transforms.Text;

namespace Samples.Dynamic
{
    public static class ProduceNgrams
    {
        public static void Example()
        {
            // Create a new ML context, for ML.NET operations. It can be used for
            // exception tracking and logging, as well as the source of randomness.
            var mlContext = new MLContext();

            // Create a small dataset as an IEnumerable.
            var samples = new List<TextData>()
            {
                new TextData(){ Text = "This is an example to compute n-grams." },
                new TextData(){ Text = "N-gram is a sequence of 'N' consecutive " +
                    "words/tokens." },

                new TextData(){ Text = "ML.NET's ProduceNgrams API produces " +
                    "vector of n-grams." },

                new TextData(){ Text = "Each position in the vector corresponds " +
                    "to a particular n-gram." },

                new TextData(){ Text = "The value at each position corresponds " +
                    "to," },

                new TextData(){ Text = "the number of times n-gram occurred in " +
                    "the data (Tf), or" },

                new TextData(){ Text = "the inverse of the number of documents " +
                    "that contain the n-gram (Idf)," },

                new TextData(){ Text = "or compute both and multiply together " +
                    "(Tf-Idf)." },
            };

            // Convert training data to IDataView.
            var dataview = mlContext.Data.LoadFromEnumerable(samples);

            // A pipeline for converting text into numeric n-gram features.
            // The following call to 'ProduceNgrams' requires the tokenized
            // text /string as input. This is achieved by calling 
            // 'TokenizeIntoWords' first followed by 'ProduceNgrams'. Please note
            // that the length of the output feature vector depends on the n-gram
            // settings.
            var textPipeline = mlContext.Transforms.Text.TokenizeIntoWords("Tokens",
                "Text")
                // 'ProduceNgrams' takes key type as input. Converting the tokens
                // into key type using 'MapValueToKey'.
                .Append(mlContext.Transforms.Conversion.MapValueToKey("Tokens"))
                .Append(mlContext.Transforms.Text.ProduceNgrams("NgramFeatures",
                    "Tokens",
                    ngramLength: 3,
                    useAllLengths: false,
                    weighting: NgramExtractingEstimator.WeightingCriteria.Tf));

            // Fit to data.
            var textTransformer = textPipeline.Fit(dataview);
            var transformedDataView = textTransformer.Transform(dataview);

            // Create the prediction engine to get the n-gram features extracted
            // from the text.
            var predictionEngine = mlContext.Model.CreatePredictionEngine<TextData,
                TransformedTextData>(textTransformer);

            // Convert the text into numeric features.
            var prediction = predictionEngine.Predict(samples[0]);

            // Print the length of the feature vector.
            Console.WriteLine("Number of Features: " + prediction.NgramFeatures
                .Length);

            // Preview of the produced n-grams.
            // Get the slot names from the column's metadata.
            // The slot names for a vector column corresponds to the names
            // associated with each position in the vector.
            VBuffer<ReadOnlyMemory<char>> slotNames = default;
            transformedDataView.Schema["NgramFeatures"].GetSlotNames(ref slotNames);
            var NgramFeaturesColumn = transformedDataView.GetColumn<VBuffer<
                float>>(transformedDataView.Schema["NgramFeatures"]);
            var slots = slotNames.GetValues();
            Console.Write("N-grams: ");
            foreach (var featureRow in NgramFeaturesColumn)
            {
                foreach (var item in featureRow.Items())
                    Console.Write($"{slots[item.Key]}  ");
                Console.WriteLine();
            }

            // Print the first 10 feature values.
            Console.Write("Features: ");
            for (int i = 0; i < 10; i++)
                Console.Write($"{prediction.NgramFeatures[i]:F4}  ");

            //  Expected output:
            //   Number of Features: 52
            //   N-grams:   This|is|an  is|an|example  an|example|to  example|to|compute  to|compute|n-grams.  N-gram|is|a  is|a|sequence  a|sequence|of  sequence|of|'N'  of|'N'|consecutive  ...
            //   Features:     1.0000      1.0000          1.0000           1.0000             1.0000            0.0000      0.0000          0.0000          0.0000          0.0000          ...
        }

        private class TextData
        {
            public string Text { get; set; }
        }

        private class TransformedTextData : TextData
        {
            public float[] NgramFeatures { get; set; }
        }
    }
}