Webcam and Device Overview

Microsoft Silverlight will reach end of support after October 2021. Learn more.

Silverlight 4 enables the ability to capture source input from audio and video devices such as webcams and microphones. This overview covers connecting and interacting with audio and video devices in your Silverlight applications.

This topic contains the following sections.

• Introducing Audio and Video Devices
• Setting up Audio and Video Devices
• Displaying Video
• Image Capture
• Capturing and Playing Audio
• Raw Video
• Privacy
• Configuration Settings
• Related Topics

Introducing Audio and Video Devices

In Silverlight 4 you can capture and display audio and video from media devices, such as webcams, TV tuner, and microphones. This enables a number of scenarios which can enhance the user experience of your Silverlight applications. Some of these scenarios are capturing and displaying images, uploading profile pictures to social networking applications, video mail and video diaries, audio mail and audio diaries, audio note taking, security cameras, augmented reality, and body gestures.

The main classes used for capturing audio and video from media devices are:

Class	Description
CaptureDeviceConfiguration	Represents a helper class for obtaining information about available capture devices (audio or video) and requesting client user permission to access the captures from available devices.
CaptureSource	Provides methods that work with specific audio or video captures from the associated capture device.
VideoCaptureDevice	Describes desired and supported video format information for a video capture device, such as a webcam.
AudioCaptureDevice	Described desired and supported formats for an audio capture device, such as a microphone.
VideoSink	Exposes the capture graph for video devices. Derive from this class to receive video information and to obtain the capture graph through VideoSink.CaptureSource.
AudioSink	Exposes the capture graph for audio devices. Derive from this class to receive audio information and to obtain the capture graph through AudioSink.CaptureSource.

Setting up Audio and Video Devices

When working with media devices in your Silverlight application, the first thing you need to do are get the available devices on the system. CaptureDeviceConfiguration exposes a number of static members for accomplishing this task. Once you have the devices, you can associate them with a CaptureSource.

CaptureSource is the main class used to interact with audio and video devices. Through the CaptureSource you can start and stop the audio and video devices, check the state of the devices, and capture a single video frame. A CaptureSource is associated with one AudioCaptureDevice and one VideoCaptureDevice, though it need not be associated with both. For instance, it is valid to have a CaptureSource that is only hooked into a VideoCaptureDevice.

AudioCaptureDevice and VideoCaptureDevice are the interfaces to the physical devices. Both of these classes implement CaptureDevice. CaptureDeviceConfiguration has a number of static methods for getting the audio and video devices on the system. GetAvailableAudioCaptureDevices returns a collection of all the audio devices on the systems and GetAvailableVideoCaptureDevices returns a collection of all the video devices on the system. GetDefaultAudioCaptureDevice will return the default audio device and GetDefaultVideoCaptureDevice will return the default video device. The default devices can be changed by the user in the Silverlight Configuration Settings dialog. See the section in this article on the Silverlight Configuration settings for more details.

On Mac OSX, if an application calls GetDefaultAudioCaptureDevice or GetDefaultVideoCaptureDevice and the default audio and video devices have not previously been set by the user, Silverlight will return the first device found. If a physical device is not connected to this CaptureDevice, an InvalidOperationException will be thrown when Start is called. This expectation should be caught by the application and the user can then be notified to manually set their default devices in the "Webcam/Mic" section of the Configuration Settings dialog.

The following example shows how to create a CaptureSource and associate it with the default audio and video devices on the system.

Dim CaptureSource As New CaptureSource()

Dim webcam = CaptureDeviceConfiguration.GetDefaultVideoCaptureDevice()

Dim audio = CaptureDeviceConfiguration.GetDefaultAudioCaptureDevice()

CaptureSource.VideoCaptureDevice = webcam
CaptureSource.AudioCaptureDevice = audio

CaptureSource captureSource = new CaptureSource();

VideoCaptureDevice webcam = CaptureDeviceConfiguration.GetDefaultVideoCaptureDevice();
AudioCaptureDevice audio = CaptureDeviceConfiguration.GetDefaultAudioCaptureDevice();          

captureSource.VideoCaptureDevice = webcam;
captureSource.AudioCaptureDevice = audio;

The following example show how a set the source of a ComboBox to the list of available video devices on the system. The FriendlyName of the device is used to bind to the control. An approach like this can be useful if you would like to give the user an easy way to choose the media device they would like to use. You can then listen to the SelectionChanged event for the control and set the CaptureSource to the appropriate device.

<ComboBox x:Name="Devices">
    <ComboBox.ItemTemplate>
        <DataTemplate>
            <TextBlock Text="{Binding FriendlyName}" />
        </DataTemplate>
    </ComboBox.ItemTemplate>
</ComboBox>

Devices.ItemsSource = CaptureDeviceConfiguration.GetAvailableAudioCaptureDevices()

Devices.ItemsSource = CaptureDeviceConfiguration.GetAvailableAudioCaptureDevices();

You can get the supported video and audio formats of a particular device by checking the SupportedFormats property. This method returns a collection of either VideoFormat or AudioFormat objects. The VideoFormat object specifies the FramesPerSecond, PixelFormat, PixelHeight, PixelWidth, and Stride. The AudioFormat object specifies the BitsPerSample, Channels, SamplesPerSecond, and the WaveFormat.

Displaying Video

To display video from a video device, such as a webcam, you must have an object to render the video onto and you must be able to start and stop the video stream.

The easiest way to solve the first problem of displaying the video feed is to use a VideoBrush. Set the source of the VideoBrush to the CaptureSource. The brush can then be used to paint a rendering object, such as a Shape like a Rectangle or even a Control like a Button.

The following example shows how to set the source of a VideoBrush to a CaptureSource and set the Fill property on a Rectangle which will be used to display the video.

<!-- This Fill property will be set to the capturedImage ImageBrush  -->
<Rectangle x:Name="webcamDisplay" />

'Brush used to paint the webcam video.
Dim webcamBrush = New VideoBrush()

' Set the source on the VideoBrush used to display the video.
webcamBrush.SetSource(CaptureSource)

' Set the Fill property on the Rectangle to the VideoBrush.
webcamDisplay.Fill = webcamBrush

// Brush used to paint the webcam video.
VideoBrush webcamBrush = new VideoBrush();

// Set the source on the VideoBrush used to display the video.
webcamBrush.SetSource(captureSource);

// Set the Fill property on the Rectangle to the VideoBrush.
webcamDisplay.Fill = webcamBrush;

The CaptureSource object solves the second problem of interacting with the device. To start the audio and video devices associated with a CaptureSource, call the Start method. To stop the audio and video devices associated with a CaptureSource, call the Stop method.

Before a Silverlight application can play audio or video form a device, it must first be granted access by the user. To request access in your Silverlight application, call the static method RequestDeviceAccess on the CaptureDeviceConfiguration class. This method must be called in response to a user initiated event, such as a Button Click event. If RequestDeviceAccess is not called from a user initiated event, the method will always return false. If the Start method is called and device access has not been granted, an InvalidOperationException will be thrown. The AllowedDeviceAccess property on CaptureDeviceConfiguration specifies whether or not device access has been granted. For more information on media devices and privacy, see the sections in this document on Privacy and Configuration Settings.

CaptureSource also contains a State property for the device. This property gets the current CaptureState of the device. The possible values for the CaptureState are Stopped, Started, and Failed.

The following example shows how to start and stop a video feed form a VideoCaptureDevice. The Start and Stop methods are called from within ButtonClick event handlers, which are user initiated events. Before the device is started, we request access to the device and we check that the device is not null. Before the device is stopped, we check that the capture state of the device is Started and we check that the device is not null.

Private Sub StartButton_Click(ByVal sender As System.Object, ByVal e As System.Windows.RoutedEventArgs)
    ' Request access to the device and verify the VideoCaptureDevice is not null.
    If CaptureDeviceConfiguration.RequestDeviceAccess _
        AndAlso (captureSource.VideoCaptureDevice IsNot Nothing) Then

        Try
            captureSource.Start()
        Catch ex As InvalidOperationException
            ' Notify user that the webcam could not be started.
            MessageBox.Show("There was a problem starting the webcam " +
                "If using a Mac, verify default devices are set correctly.  " +
                "Right click on a Silverlight app to access the Configuration setings.")
        End Try

    End If
End Sub

Private Sub StopButton_Click(ByVal sender As System.Object, ByVal e As System.Windows.RoutedEventArgs)
    ' Verify the VideoCaptureDevice is not null.
    If (captureSource.VideoCaptureDevice IsNot Nothing) Then
        captureSource.Stop()
    End If
End Sub

private void StartButton_Click(object sender, RoutedEventArgs e)
{
    // Request access to the device and verify the VideoCaptureDevice is not null.
    if (CaptureDeviceConfiguration.RequestDeviceAccess() && captureSource.VideoCaptureDevice != null)
    {
        try
        {
            captureSource.Start();
        }
        catch (InvalidOperationException ex)
        {
            // Notify user that the webcam could not be started.
            MessageBox.Show("There was a problem starting the webcam " +
                "If using a Mac, verify default devices are set correctly.  " +
                "Right click on a Silverlight app to access the Configuration setings.");
        }
    }

    if (CaptureDeviceConfiguration.RequestDeviceAccess() && captureSource.VideoCaptureDevice != null)
    {
        try
        {
            captureSource.Start();
        }
        catch (InvalidOperationException ex)
        {

        }
    }
}

private void StopButton_Click(object sender, RoutedEventArgs e)
{
    // Verify the VideoCaptureDevice is not null.
    if (captureSource.VideoCaptureDevice != null)
    {
        captureSource.Stop();
    }
}

Image Capture

You can capture individual frames from a video device by using the CaptureImageAsync method on the CaptureSource. This method initiates an asynchronous image capture. To be notified when then image capture is complete and to get a handle to the captured image, listen to the CaptureImageCompleted event. To be notified if an image capture failed, listen to the CaptureFailed event.

CaptureImageCompletedEventArgs is the event argument class for CaptureImageCompleted. The Result property on the argument object is the captured image. The image is returned as a WriteableBitmap. This WriteableBitmap can then be displayed, saved, or modified. Two other properties on the event arguments that may be of interest are Cancelled and Error. Cancelled is a boolean value which specifies whether the asynchronous operation was canceled. The Error property is an exception that indicates the error which occurred during the asynchronous operation. If no error occurred, Error is null.

The following example creates a Rectangle and an ImageBrush to display the captured image.

<!-- This Fill property will be set to the capturedImage ImageBrush  -->
<Rectangle x:Name="capturedDisplay" />

' Create the ImageBrush to store the captured image.
Dim capturedImage = New ImageBrush()

' Paint the Rectangle with the ImageBrush.
capturedDisplay.Fill = capturedImage

// Create the ImageBrush to store the captured image.
ImageBrush capturedImage = new ImageBrush();

// Paint the Rectangle with the ImageBrush.
capturedDisplay.Fill = capturedImage;

The following example shows how to initiate an asynchronous image capture. Before the image capture is initiated, we first check that the device is not null and that the state of the device is started.

Private Sub CaptureButton_Click(ByVal sender As System.Object, ByVal e As System.Windows.RoutedEventArgs)
    'Verify the VideoCaptureDevice is not null and the device is started.
    If (captureSource.VideoCaptureDevice IsNot Nothing) _
        AndAlso (captureSource.State = CaptureState.Started) Then

        captureSource.CaptureImageAsync()

    End If
End Sub

private void CaptureButton_Click(object sender, RoutedEventArgs e)
{
    // Verify the VideoCaptureDevice is not null and the device is started.
    if (captureSource.VideoCaptureDevice != null && captureSource.State == CaptureState.Started)
    {
        captureSource.CaptureImageAsync();
    }
}

The following example creates event handlers for the CaptureImageCompleted event and the CaptureFailed event. In the CaptureImageCompleted event handler, the ImageSource property on the ImageBrush is set to the Result property.

AddHandler CaptureSource.CaptureImageCompleted, AddressOf CaptureSource_CaptureImageCompleted

AddHandler CaptureSource.CaptureFailed, AddressOf CaptureSource_CaptureFailed

captureSource.CaptureImageCompleted +=new 
    EventHandler<CaptureImageCompletedEventArgs>(CaptureSource_CaptureImageCompleted);

captureSource.CaptureFailed += 
    new EventHandler<ExceptionRoutedEventArgs>(CaptureSource_CaptureFailed);

Private Sub CaptureSource_CaptureImageCompleted(ByVal sender As Object, ByVal e As CaptureImageCompletedEventArgs)
    ' Set the ImageBrush source (defined in XAML) to the Result property
    capturedImage.ImageSource = e.Result
End Sub

Private Sub CaptureSource_CaptureFailed(ByVal sender As Object, ByVal e As ExceptionRoutedEventArgs)
    ' Handle the failure.
End Sub

void CaptureSource_CaptureImageCompleted(object sender, CaptureImageCompletedEventArgs e)
{            
    // Set the ImageBrush source (defined in XAML) to the Result property
    capturedImage.ImageSource = e.Result;
}

void CaptureSource_CaptureFailed(object sender, ExceptionRoutedEventArgs e)
{
    // Handle the failure.
}

Capturing and Playing Audio

To capture and play back audio, you will need to use an AudioSink object. This object gives you access to the raw audio stream coming from the audio device. This stream can then be encoded into a common format. AudioSink is an abstract class so you will need to create a class that implements AudioSink. There are four virtual callbacks you need to override: OnCaptureStarted, OnCaptureStopped, OnFormatChange, and OnSamples.

OnCaptureStarted and OnCaptureStopped occur when the audio device is started and stopped. These notifications can be useful for creating sentinels in your application to block certain actions when capture is occurring.

OnFormatChange occurs when the devices detects the audio format has changed. OnFormatChange is invoked at least one time for the first capture by the sink. Therefore, you can use OnFormatChange as the definitive determination of the audio format instead of using DesiredFormat or SupportedFormats. It is possible for the AudioFormat associated with the AudioSink to change between the time that the device was first connected and the time of the capture. Listening to OnFormatChange is the only way to be certain of the true AudioFormat.

The audio format is passed into OnFormatChange as an AudioFormat object.

OnSamples is invoked when the audio device captures a complete audio sample. The arguments passed into this method are the sample time (in hundreds of nanoseconds), the sample duration (in hundred of nanoseconds), and the sample data as a byte array. The interval at which OnSamples is called depends on the AudioFrameSize value for the AudioCaptureDevice. The AudioFormat specifies the bits per second, the number of channels, and the wave format. This information can then be use to convert the sample data into the desired format.

Raw Video

Just as you can get access to the raw audio from the audio device, you can also get the raw video form the video device through a VideoSink. VideoSink is an abstract class so you will need to create a class that implements VideoSink. The structure of the VideoSink is similar to the AudioSink. You will need to override the same four virtual callbacks: OnCaptureStarted, OnCaptureStopped, OnFormatChange, and OnSample.

These methods behave in a similar manner as their AudioSink counter parts.

In OnSample you can use the information from the VideoFormat, which can be obtained by listening to the OnFormatChange method, to convert the bit stream into the desired video format. The VideoFormat specifies the pixel format, the height of the frame in pixels, the width of the frame in pixels, and the stride.

Privacy

In order for Silverlight to gain access to audio and video devices, the user of the application must first grant access to the devices. The user is prompted with a privacy dialog asking if they want to allow camera and microphone access. If the user agrees to this dialog, then the Silverlight application is granted access to the devices; otherwise, the application cannot access the devices. On the privacy dialog there is a check box for remembering the choice the user made. Access rights can be changed, removed, or added for sites in the Silverlight Configuration dialog (right click on a running Silverlight application).

Configuration Settings

The Silverlight Configuration Settings dialog has a few options that pertain to audio and video devices. To open the configuration settings, right click on any running Silverlight application and choose the Silverlight node. For more information on the Silverlight Configuration Settings, see the Microsoft Silverlight Configuration Dialog Box topic.

The Configuration Settings has a node for webcams and microphones. From this dialog you can change the default video device and the default audio device. These settings are independent of each other, so you could choose video from one device and audio from another device.

From the Permission tab on the Configuration Settings dialog you can allow, deny, or remove access to a particular site. If for example, a user granted webcam access to a particular site they could remove that access from this dialog.

See Also

Reference

Microsoft Silverlight Configuration Dialog Box

CaptureSource

VideoCaptureDevice

AudioCaptureDevice