When using AutoML for training a Win32Exception arises indicating that the wait operation timed out after 30 seconds

Andreas ss 726 Reputation points
2024-06-03T18:31:48.42+00:00

CPU: i7-12800h (14 cores, Total Threads: 20)

RAM: 32 GB SSD: Samsung 980 Pro, 2 TB

Windows Server 2019 Datacenter Evaluation

NET Framework: 4.8

Microsoft.ML: 3.0.1

Microsoft.ML.AutoML: 0.21.1

Issue: When I start to train a model using AutoML, I get a timeout after exactly 30 seconds and the application breaks/stops with below error message: (CPU working at 15%. RAM has 50% avaliable memory at the exception)

System.AggregateException: 'One or more errors occurred.' TargetInvocationException: Exception has been thrown by the target of an invocation. SqlException: Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding. Win32Exception: The wait operation timed out

Normally, the training of the Model should continue but I don't understand why I get this timeout after 30 seconds. It seems to be a default value and this must be possible to increase and change but I have not found out where to change this default value or how to solve this problem?

IDataViews are already loaded before the Experiment and looks like this:
trainData: Rows: 384846, Columns: 64
hold_out_data: Rows: 33958, Columns: 64

Below is the code I use:
(You might mention to use "MaxModels" but the root problem is about be able to change the timeout period, I beleive)

        void trainer_function(IDataView trainData, IDataView hold_out_data)
        {
            MLContext mlContext = new MLContext();
            var experiment = mlContext.Auto().CreateBinaryClassificationExperiment(new BinaryExperimentSettings
            {
                MaxExperimentTimeInSeconds = 600,
                CacheBeforeTrainer = CacheBeforeTrainer.On,
                CacheDirectoryName = "C:/Aintelligence/temp/cache",
                MaximumMemoryUsageInMegaByte = 8192,
                OptimizingMetric = BinaryClassificationMetric.PositivePrecision,
                CancellationToken = CancellationToken.None
            });
            var progressHandler = new Progress<RunDetail<BinaryClassificationMetrics>>(ph =>
            {
                if (ph.ValidationMetrics != null && !ph.TrainerName.Contains("FastForest"))
                {
                    double positivePrecision = Math.Round(ph.ValidationMetrics.PositivePrecision, 3); //Do something with: "positivePrecision"
                }
            });
            //Start the experiment/Training
            var results = experiment.Execute(trainData, hold_out_data, labelColumnName: "Label", progressHandler: progressHandler);
        }

SQL Server
SQL Server
A family of Microsoft relational database management and analysis systems for e-commerce, line-of-business, and data warehousing solutions.
14,223 questions
C#
C#
An object-oriented and type-safe programming language that has its roots in the C family of languages and includes support for component-oriented programming.
11,156 questions
.NET Machine learning
.NET Machine learning
.NET: Microsoft Technologies based on the .NET software framework.Machine learning: A type of artificial intelligence focused on enabling computers to use observed data to evolve new behaviors that have not been explicitly programmed.
159 questions
0 comments No comments
{count} votes

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.