about_Thread_Jobs
Short description
Provides information about PowerShell thread-based jobs. A thread job is a type of background job that runs a command or expression in a separate thread within the current session process.
Long description
PowerShell concurrently runs commands and scripts through jobs. There are three jobs types provided by PowerShell to support concurrency.
RemoteJob- Commands and scripts run in a remote session. For information, see about_Remote_Jobs.BackgroundJob- Commands and scripts run in a separate process on the local machine. For more information, see about_Jobs.PSTaskJoborThreadJob- Commands and scripts run in a separate thread within the same process on the local machine.
Thread-based jobs are not as robust as remote and background jobs, because they run in the same process on different threads. If one job has a critical error that crashes the process, then all other jobs in the process are terminated.
However, thread-based jobs require less overhead. They don't use the remoting layer or serialization. The result objects are returned as references to live objects in the current session. Without this overhead, thread-based jobs run faster and use fewer resources than the other job types.
Important
The parent session that created the job also monitors the job status and collects pipeline data. The job child process is terminated by the parent process once the job reaches a finished state. If the parent session is terminated, all running child jobs are terminated along with their child processes.
There are two ways work around this situation:
- Use
Invoke-Commandto create jobs that run in disconnected sessions. For more information, see about_Remote_Jobs. - Use
Start-Processto create a new process rather than a job. For more information, see Start-Process.
How to start and manage thread-based jobs
There are two ways to start thread-based jobs:
Start-ThreadJob- from the ThreadJob moduleForEach-Object -Parallel -AsJob- the parallel feature was added in PowerShell 7.0
Use the same Job cmdlets described in about_Jobs to manage thread-based jobs.
Using Start-ThreadJob
The ThreadJob module first shipped with PowerShell 6. It can also be installed from the PowerShell Gallery for Windows PowerShell 5.1.
To start a thread job on the local computer, use the Start-ThreadJob cmdlet
with a command or script enclosed in curly braces ({ }).
The following example starts a thread job that runs a Get-Process command on
the local computer.
Start-ThreadJob -ScriptBlock { Get-Process }
The Start-ThreadJob command returns a ThreadJob object that represents the
running job. The job object contains useful information about the job including
its current running status. It collects the results of the job as the results
are being generated.
Using ForEach-Object -Parallel -AsJob
PowerShell 7.0 added a new parameter set to the ForEach-Object cmdlet. The
new parameters allow you to run script blocks in parallel threads as PowerShell
jobs.
You can pipe data to ForEach-Object -Parallel. The data is passed to the
script block that is run in parallel. The -AsJob parameter creates jobs
objects for each of the parallel threads.
The following command starts a job that contains child jobs for each input value
piped to the command. Each child job runs the Write-Output command with a
piped input value as the argument.
1..5 | ForEach-Object -Parallel { Write-Output $_ } -AsJob
The ForEach-Object -Parallel command returns a PSTaskJob object that
contains child jobs for each piped input value. The job object contains useful
information about the child jobs running status. It collects the results of the
child jobs as the results are being generated.
How to wait for a job to complete and retrieve job results
You can use PowerShell job cmdlets, such as Wait-Job and Receive-Job to
wait for a job to complete and then return all results generated by the job.
The following command starts a thread job that runs a Get-Process command,
then waits for the command to complete, and finally returns all data results
generated by the command.
Start-ThreadJob -ScriptBlock { Get-Process } | Wait-Job | Receive-Job
The following command starts a job that runs a Write-Output command for each
piped input, then waits for all child jobs to complete, and finally returns all
data results generated by the child jobs.
1..5 | ForEach-Object -Parallel { Write-Output $_ } -AsJob | Wait-Job | Receive-Job
The Receive-Job cmdlet returns the results of the child jobs.
1
3
2
4
5
Because each child job runs parallel, the order of the generated results is not guaranteed.
Thread job performance
Thread jobs are faster and lighter weight than other types of jobs. But they still have overhead that can be large when compared to work the job is doing.
PowerShell runs commands and script in a session. Only one command or script can run at a time in a session. So when running multiple jobs, each job runs in a separate session. Each session contributes to the overhead.
Thread jobs provide the best performance when the work they perform is greater than the overhead of the session used to run the job. There are two cases for that meet this criteria.
Work is compute intensive - Running a script on multiple thread jobs can take advantage of multiple processor cores and complete faster.
Work consists of significant waiting - A script that spends time waiting for I/O or remote call results. Running in parallel usually completes quicker than if run sequentially.
(Measure-Command {
1..1000 | ForEach { Start-ThreadJob { Write-Output "Hello $using:_" } } | Receive-Job -Wait
}).TotalMilliseconds
36860.8226
(Measure-Command {
1..1000 | ForEach-Object { "Hello: $_" }
}).TotalMilliseconds
7.1975
The first example above shows a foreach loop that creates 1000 thread jobs to do a simple string write. Due to job overhead, it takes over 36 seconds to complete.
The second example runs the ForEach cmdlet to do the same 1000 operations.
This time, ForEach-Object runs sequentially, on a single thread, without any
job overhead. It completes in a mere 7 milliseconds.
In the following example, up to 5000 entries are collected for 10 separate system logs. Since the script involves reading a number of logs, it makes sense to do the operations in parallel.
$logNames.count
10
Measure-Command {
$logs = $logNames | ForEach-Object {
Get-WinEvent -LogName $_ -MaxEvents 5000 2>$null
}
}
TotalMilliseconds : 252398.4321 (4 minutes 12 seconds)
$logs.Count
50000
The script completes in half the time when the jobs are run in parallel.
Measure-Command {
$logs = $logNames | ForEach {
Start-ThreadJob {
Get-WinEvent -LogName $using:_ -MaxEvents 5000 2>$null
} -ThrottleLimit 10
} | Wait-Job | Receive-Job
}
TotalMilliseconds : 115994.3 (1 minute 56 seconds)
$logs.Count
50000
Thread jobs and variables
There are multiple ways to pass values into the thread-based jobs.
Start-ThreadJob can accept variables that are piped to the cmdlet, passed in
to the script block via the $using keyword, or passed in via the
ArgumentList parameter.
$msg = "Hello"
$msg | Start-ThreadJob { $input | Write-Output } | Wait-Job | Receive-Job
Start-ThreadJob { Write-Output $using:msg } | Wait-Job | Receive-Job
Start-ThreadJob { param ([string] $message) Write-Output $message } -ArgumentList @($msg) |
Wait-Job | Receive-Job
ForEach-Object -Parallel accepts piped in variables, and variables passed
directly to the script block via the $using keyword.
$msg = "Hello"
$msg | ForEach-Object -Parallel { Write-Output $_ } -AsJob | Wait-Job | Receive-Job
1..1 | ForEach-Object -Parallel { Write-Output $using:msg } -AsJob | Wait-Job | Receive-Job
Since thread jobs run in the same process, any variable reference type passed into the job has to be treated carefully. If it is not a thread safe object, then it should never be assigned to, and method and properties should never be invoked on it.
The following example passes a thread-safe .NET ConcurrentDictionary object
to all child jobs to collect uniquely named process objects. Since it is a
thread safe object, it can be safely used while the jobs run concurrently in
the process.
$threadSafeDictionary = [System.Collections.Concurrent.ConcurrentDictionary[string,object]]::new()
$jobs = Get-Process | ForEach {
Start-ThreadJob {
$proc = $using:_
$dict = $using:threadSafeDictionary
$dict.TryAdd($proc.ProcessName, $proc)
}
}
$jobs | Wait-Job | Receive-Job
$threadSafeDictionary.Count
96
$threadSafeDictionary["pwsh"]
NPM(K) PM(M) WS(M) CPU(s) Id SI ProcessName
------ ----- ----- ------ -- -- -----------
112 108.25 124.43 69.75 16272 1 pwsh
