Media compositions and editing

Article
10/20/2022

This article shows you how to use the APIs in the Windows.Media.Editing namespace to quickly develop apps that enable the users to create media compositions from audio and video source files. Features of the framework include the ability to programmatically append multiple video clips together, add video and image overlays, add background audio, and apply both audio and video effects. Once created, media compositions can be rendered into a flat media file for playback or sharing, but compositions can also be serialized to and deserialized from disk, allowing the user to load and modify compositions that they have previously created. All of this functionality is provided in an easy-to-use Windows Runtime interface that dramatically reduces the amount and complexity of code required to perform these tasks when compared to the low-level Microsoft Media Foundation API.

Create a new media composition

The MediaComposition class is the container for all of the media clips that make up the composition and is responsible for rendering the final composition, loading and saving compositions to disc, and providing a preview stream of the composition so that the user can view it in the UI. To use MediaComposition in your app, include the Windows.Media.Editing namespace as well as the Windows.Media.Core namespace that provides related APIs that you will need.

using Windows.Media.Editing;
using Windows.Media.Core;
using Windows.Media.Playback;
using System.Threading.Tasks;

The MediaComposition object will be accessed from multiple points in your code, so typically you will declare a member variable in which to store it.

private MediaComposition composition;

The constructor for MediaComposition takes no arguments.

composition = new MediaComposition();

Add media clips to a composition

Media compositions typically contain one or more video clips. You can use a FileOpenPicker to allow the user to select a video file. Once the file has been selected, create a new MediaClip object to contain the video clip by calling MediaClip.CreateFromFileAsync. Then you add the clip to the MediaComposition object's Clips list.

private async Task PickFileAndAddClip()
{
    var picker = new Windows.Storage.Pickers.FileOpenPicker();
    picker.SuggestedStartLocation = Windows.Storage.Pickers.PickerLocationId.VideosLibrary;
    picker.FileTypeFilter.Add(".mp4");
    Windows.Storage.StorageFile pickedFile = await picker.PickSingleFileAsync();
    if (pickedFile == null)
    {
        ShowErrorMessage("File picking cancelled");
        return;
    }

    // These files could be picked from a location that we won't have access to later
    var storageItemAccessList = Windows.Storage.AccessCache.StorageApplicationPermissions.FutureAccessList;
    storageItemAccessList.Add(pickedFile);

    var clip = await MediaClip.CreateFromFileAsync(pickedFile);
    composition.Clips.Add(clip);

}

Media clips appear in the MediaComposition in the same order as they appear in Clips list.
A MediaClip can only be included in a composition once. Attempting to add a MediaClip that is already being used by the composition will result in an error. To reuse a video clip multiple times in a composition, call Clone to create new MediaClip objects which can then be added to the composition.
Universal Windows apps do not have permission to access the entire file system. The FutureAccessList property of the StorageApplicationPermissions class allows your app to store a record of a file that has been selected by the user so that you can retain permissions to access the file. The FutureAccessList has a maximum of 1000 entries, so your app needs to manage the list to make sure it does not become full. This is especially important if you plan to support loading and modifying previously created compositions.
A MediaComposition supports video clips in MP4 format.
If a video file contains multiple embedded audio tracks, you can select which audio track is used in the composition by setting the SelectedEmbeddedAudioTrackIndex property.
Create a MediaClip with a single color filling the entire frame by calling CreateFromColor and specifying a color and a duration for the clip.
Create a MediaClip from an image file by calling CreateFromImageFileAsync and specifying an image file and a duration for the clip.
Create a MediaClip from an IDirect3DSurface by calling CreateFromSurface and specifying a surface and a duration from the clip.

Preview the composition in a MediaElement

To enable the user to view the media composition, add a MediaPlayerElement to the XAML file that defines your UI.

<MediaPlayerElement x:Name="mediaPlayerElement" AutoPlay="False" Margin="5" HorizontalAlignment="Stretch" AreTransportControlsEnabled="True" />

Declare a member variable of type MediaStreamSource.

private MediaStreamSource mediaStreamSource;

Call the MediaComposition object's GeneratePreviewMediaStreamSource method to create a MediaStreamSource for the composition. Create a MediaSource object by calling the factory method CreateFromMediaStreamSource and assign it to the Source property of the MediaPlayerElement. Now the composition can be viewed in the UI.

public void UpdateMediaElementSource()
{

    mediaStreamSource = composition.GeneratePreviewMediaStreamSource(
        (int)mediaPlayerElement.ActualWidth,
        (int)mediaPlayerElement.ActualHeight);

    mediaPlayerElement.Source = MediaSource.CreateFromMediaStreamSource(mediaStreamSource);

}

The MediaComposition must contain at least one media clip before calling GeneratePreviewMediaStreamSource, or the returned object will be null.
The MediaElement timeline is not automatically updated to reflect changes in the composition. It is recommended that you call both GeneratePreviewMediaStreamSource and set the MediaPlayerElement Source property every time you make a set of changes to the composition and want to update the UI.

It is recommended that you set the MediaStreamSource object and the Source property of the MediaPlayerElement to null when the user navigates away from the page in order to release associated resources.

protected override void OnNavigatedFrom(NavigationEventArgs e)
{
    mediaPlayerElement.Source = null;
    mediaStreamSource = null;
    base.OnNavigatedFrom(e);

}

Render the composition to a video file

To render a media composition to a flat video file so that it can be shared and viewed on other devices, you will need to use APIs from the Windows.Media.Transcoding namespace. To update the UI on the progress of the async operation, you will also need APIs from the Windows.UI.Core namespace.

using Windows.Media.Transcoding;
using Windows.UI.Core;

After allowing the user to select an output file with a FileSavePicker, render the composition to the selected file by calling the MediaComposition object's RenderToFileAsync. The rest of the code in the following example simply follows the pattern of handling an AsyncOperationWithProgress.

private async Task RenderCompositionToFile()
{
    var picker = new Windows.Storage.Pickers.FileSavePicker();
    picker.SuggestedStartLocation = Windows.Storage.Pickers.PickerLocationId.VideosLibrary;
    picker.FileTypeChoices.Add("MP4 files", new List<string>() { ".mp4" });
    picker.SuggestedFileName = "RenderedComposition.mp4";

    Windows.Storage.StorageFile file = await picker.PickSaveFileAsync();
    if (file != null)
    {
        // Call RenderToFileAsync
        var saveOperation = composition.RenderToFileAsync(file, MediaTrimmingPreference.Precise);

        saveOperation.Progress = new AsyncOperationProgressHandler<TranscodeFailureReason, double>(async (info, progress) =>
        {
            await this.Dispatcher.RunAsync(CoreDispatcherPriority.Normal, new DispatchedHandler(() =>
            {
                ShowErrorMessage(string.Format("Saving file... Progress: {0:F0}%", progress));
            }));
        });
        saveOperation.Completed = new AsyncOperationWithProgressCompletedHandler<TranscodeFailureReason, double>(async (info, status) =>
        {
            await this.Dispatcher.RunAsync(CoreDispatcherPriority.Normal, new DispatchedHandler(() =>
            {
                try
                {
                    var results = info.GetResults();
                    if (results != TranscodeFailureReason.None || status != AsyncStatus.Completed)
                    {
                        ShowErrorMessage("Saving was unsuccessful");
                    }
                    else
                    {
                        ShowErrorMessage("Trimmed clip saved to file");
                    }
                }
                finally
                {
                        // Update UI whether the operation succeeded or not
                    }

            }));
        });
    }
    else
    {
        ShowErrorMessage("User cancelled the file selection");
    }
}

The MediaTrimmingPreference allows you to prioritize speed of the transcoding operation versus the precision of trimming of adjacent media clips. Fast causes transcoding to be faster with lower-precision trimming, Precise causes transcoding to be slower but with more precise trimming.

Trim a video clip

Trim the duration of a video clip in a composition by setting the MediaClip objects TrimTimeFromStart property, the TrimTimeFromEnd property, or both.

private void TrimClipBeforeCurrentPosition()
{
    var currentClip = composition.Clips.FirstOrDefault(
        mc => mc.StartTimeInComposition <= mediaPlayerElement.MediaPlayer.PlaybackSession.Position &&
        mc.EndTimeInComposition >= mediaPlayerElement.MediaPlayer.PlaybackSession.Position);

    TimeSpan positionFromStart = mediaPlayerElement.MediaPlayer.PlaybackSession.Position - currentClip.StartTimeInComposition;
    currentClip.TrimTimeFromStart = positionFromStart;

}

You can use any UI that you want to let the user specify the start and end trim values. The example above uses the Position property of the MediaPlaybackSession associated with the MediaPlayerElement to first determine which MediaClip is playing back at the current position in the composition by checking the StartTimeInComposition and EndTimeInComposition. Then the Position and StartTimeInComposition properties are used again to calculate the amount of time to trim from the beginning of the clip. The FirstOrDefault method is an extension method from the System.Linq namespace that simplifies the code for selecting items from a list.
The OriginalDuration property of the MediaClip object lets you know the duration of the media clip without any clipping applied.
The TrimmedDuration property lets you know the duration of the media clip after trimming is applied.
Specifying a trimming value that is larger than the original duration of the clip does not throw an error. However, if a composition contains only a single clip and that is trimmed to zero length by specifying a large trimming value, a subsequent call to GeneratePreviewMediaStreamSource will return null, as if the composition has no clips.

Add a background audio track to a composition

To add a background track to a composition, load an audio file and then create a BackgroundAudioTrack object by calling the factory method BackgroundAudioTrack.CreateFromFileAsync. Then, add the BackgroundAudioTrack to the composition's BackgroundAudioTracks property.

private async Task AddBackgroundAudioTrack()
{
    // Add background audio
    var picker = new Windows.Storage.Pickers.FileOpenPicker();
    picker.SuggestedStartLocation = Windows.Storage.Pickers.PickerLocationId.MusicLibrary;
    picker.FileTypeFilter.Add(".mp3");
    picker.FileTypeFilter.Add(".wav");
    picker.FileTypeFilter.Add(".flac");
    Windows.Storage.StorageFile audioFile = await picker.PickSingleFileAsync();
    if (audioFile == null)
    {
        ShowErrorMessage("File picking cancelled");
        return;
    }

    // These files could be picked from a location that we won't have access to later
    var storageItemAccessList = Windows.Storage.AccessCache.StorageApplicationPermissions.FutureAccessList;
    storageItemAccessList.Add(audioFile);

    var backgroundTrack = await BackgroundAudioTrack.CreateFromFileAsync(audioFile);

    composition.BackgroundAudioTracks.Add(backgroundTrack);

}

A MediaComposition supports background audio tracks in the following formats: MP3, WAV, FLAC
A background audio track
As with video files, you should use the StorageApplicationPermissions class to preserve access to files in the composition.
As with MediaClip, a BackgroundAudioTrack can only be included in a composition once. Attempting to add a BackgroundAudioTrack that is already being used by the composition will result in an error. To reuse an audio track multiple times in a composition, call Clone to create new MediaClip objects which can then be added to the composition.
By default, background audio tracks begin playing at the start of the composition. If multiple background tracks are present, all of the tracks will begin playing at the start of the composition. To cause a background audio track to be begin playback at another time, set the Delay property to the desired time offset.

Add an overlay to a composition

Overlays allow you to stack multiple layers of video on top of each other in a composition. A composition can contain multiple overlay layers, each of which can include multiple overlays. Create a MediaOverlay object by passing a MediaClip into its constructor. Set the position and opacity of the overlay, then create a new MediaOverlayLayer and add the MediaOverlay to its Overlays list. Finally, add the MediaOverlayLayer to the composition's OverlayLayers list.

private void AddOverlay(MediaClip overlayMediaClip, double scale, double left, double top, double opacity)
{
    Windows.Media.MediaProperties.VideoEncodingProperties encodingProperties =
        overlayMediaClip.GetVideoEncodingProperties();

    Rect overlayPosition;

    overlayPosition.Width = (double)encodingProperties.Width * scale;
    overlayPosition.Height = (double)encodingProperties.Height * scale;
    overlayPosition.X = left;
    overlayPosition.Y = top;

    MediaOverlay mediaOverlay = new MediaOverlay(overlayMediaClip);
    mediaOverlay.Position = overlayPosition;
    mediaOverlay.Opacity = opacity;

    MediaOverlayLayer mediaOverlayLayer = new MediaOverlayLayer();
    mediaOverlayLayer.Overlays.Add(mediaOverlay);

    composition.OverlayLayers.Add(mediaOverlayLayer);
}

Overlays within a layer are z-ordered based on their order in their containing layer's Overlays list. Higher indices within the list are rendered on top of lower indices. The same is true of overlay layers within a composition. A layer with higher index in the composition's OverlayLayers list will be rendered on top of lower indices.
Because overlays are stacked on top of each other instead of being played sequentially, all overlays start playback at the beginning of the composition by default. To cause an overlay to be begin playback at another time, set the Delay property to the desired time offset.

Add effects to a media clip

Each MediaClip in a composition has a list of audio and video effects to which multiple effects can be added. The effects must implement IAudioEffectDefinition and IVideoEffectDefinition respectively. The following example uses the current MediaPlayerElement position to choose the currently viewed MediaClip and then creates a new instance of the VideoStabilizationEffectDefinition and appends it to the media clip's VideoEffectDefinitions list.

private void AddVideoEffect()
{
    var currentClip = composition.Clips.FirstOrDefault(
        mc => mc.StartTimeInComposition <= mediaPlayerElement.MediaPlayer.PlaybackSession.Position &&
        mc.EndTimeInComposition >= mediaPlayerElement.MediaPlayer.PlaybackSession.Position);

    VideoStabilizationEffectDefinition videoEffect = new VideoStabilizationEffectDefinition();
    currentClip.VideoEffectDefinitions.Add(videoEffect);
}

Save a composition to a file

Media compositions can be serialized to a file to be modified at a later time. Pick an output file and then call the MediaComposition method SaveAsync to save the composition.

private async Task SaveComposition()
{
    var picker = new Windows.Storage.Pickers.FileSavePicker();
    picker.SuggestedStartLocation = Windows.Storage.Pickers.PickerLocationId.VideosLibrary;
    picker.FileTypeChoices.Add("Composition files", new List<string>() { ".cmp" });
    picker.SuggestedFileName = "SavedComposition";

    Windows.Storage.StorageFile compositionFile = await picker.PickSaveFileAsync();
    if (compositionFile == null)
    {
        ShowErrorMessage("User cancelled the file selection");
    }
    else
    {
        var action = composition.SaveAsync(compositionFile);
        action.Completed = (info, status) =>
        {
            if (status != AsyncStatus.Completed)
            {
                ShowErrorMessage("Error saving composition");
            }

        };
    }
}

Load a composition from a file

Media compositions can be deserialized from a file to allow the user to view and modify the composition. Pick a composition file and then call the MediaComposition method LoadAsync to load the composition.

private async Task OpenComposition()
{
    var picker = new Windows.Storage.Pickers.FileOpenPicker();
    picker.SuggestedStartLocation = Windows.Storage.Pickers.PickerLocationId.VideosLibrary;
    picker.FileTypeFilter.Add(".cmp");

    Windows.Storage.StorageFile compositionFile = await picker.PickSingleFileAsync();
    if (compositionFile == null)
    {
        ShowErrorMessage("File picking cancelled");
    }
    else
    {
        composition = null;
        composition = await MediaComposition.LoadAsync(compositionFile);

        if (composition != null)
        {
            UpdateMediaElementSource();

        }
        else
        {
            ShowErrorMessage("Unable to open composition");
        }
    }
}

If a media file in the composition is not in a location that can be accessed by your app and is not in the FutureAccessList property of the StorageApplicationPermissions class for your app, an error will be thrown when loading the composition.

Share via