Redigera

Dela via

Media Sinks

  • Artikel

Media sinks are the pipeline objects that receive media data. A media sink is the destination for one or more media streams. Media sinks fall into two general categories:

  • A renderer is a media sink that presents data for playback. The enhanced video renderer (EVR) displays video frames, and the audio renderer plays audio streams through the sound card or other audio device.

  • An archive sink is a media sink that writes data to a file or other storage.

The main difference between them is that an archive sink does not consume the data at a fixed playback rate. Instead, it writes the data that it receives as quickly as possible.

Media sinks expose the IMFMediaSink interface. Each media sink contains one or more stream sinks. Each stream sink receives the data from one stream. Stream sinks expose the IMFStreamSink interface. Typically an application does not create media sinks directly. Instead, the application creates one or more activation objects, which the Media Session uses to create the sink. All other operations on the sink are handled by the Media Session, and the application does not call any methods on the media sink or any of the stream sinks. For more information about activation objects, see Activation Objects.

You should read the remainder of this topic if you are writing a custom media sink, or if you want to use a media sink directly without the Media Session.

Stream Sinks

A media sink can have a fixed number of stream sinks, or it can support adding and removing stream sinks. If it has a fixed number of stream sinks, the IMFMediaSink::GetCharacteristics method returns the MEDIASINK_FIXED_STREAMS flag. Otherwise, you can add and remove stream sinks. To add a new stream sink, call IMFMediaSink::AddStreamSink. To remove a stream sink, call IMFMediaSink::RemoveStreamSink. The MEDIASINK_FIXED_STREAMS flag indicates that the media sink does not support these two methods. (It might support some other way to configure the number of streams, for example by setting initialization parameters when the sink is created.) The list of stream sinks is ordered. To enumerate them by index value, call the IMFMediaSink::GetStreamSinkByIndex method.

Stream sinks also have identifiers. Stream identifiers are unique within the media sink, but do not have to be consecutive. Depending on the media sink, the stream identifiers might have some meaning that is related to the content. For example, an archive sink might write the stream identifiers into the file header. Otherwise, they are arbitrary. To get a stream sink by its identifier, call IMFMediaSink::GetStreamSinkById.

Presentation Clock

The rate at which a media sink consumes samples is controlled by the Presentation Clock. The Media Session selects the presentation clock and sets it on the media sink by calling the media sink's IMFMediaSink::SetPresentationClock method. The presentation clock must be set on the media sink before streaming can begin. Every media sink requires a presentation clock to run. The media sink uses the presentation clock for two purposes:

  • To receive notifications when streaming starts or stops. The media sink receives these notifications through the IMFClockStateSink interface, which all media sinks must implement.

  • To determine when it should render samples. When the media sink receives a new sample, it gets the time stamp from the sample and tries to render the sample at that presentation time.

The presentation clock derives its clock times from another object called the presentation time source. Presentation time sources expose the IMFPresentationTimeSource interface. Some media sinks have access to an accurate clock, so they expose this interface. This means that a media sink might schedule samples against a time provided by its own clock. However, the media sink cannot assume this to be the case. It must always use the time from the presentation clock, regardless of whether the presentation clock is driven by the media sink itself, or by some other clock.

If a media sink cannot match rates with a clock other than its own, the GetCharacteristics method returns the MEDIASINK_CANNOT_MATCH_CLOCK flag. If this flag is present and the presentation clock uses a different presentation time source, the media sink is likely to perform poorly. For example, it might glitch during playback.

A rateless sink is a media sink that ignores the time stamps on samples and consumes data as soon as each sample arrives. A rateless media sink returns the MEDIASINK_RATELESS flag from the GetCharacteristics method. Typically this flag applies to archive sinks. If every media sink in the pipeline is rateless, the Media Session uses a special rateless presentation clock. This clock runs as quickly as the sinks consume samples.

Stream Formats

Before the media sink can receive samples, the client must set the media type on the stream sinks. To set the media type, call the stream sink's IMFStreamSink::GetMediaTypeHandler method. This method returns a pointer to the IMFMediaTypeHandler interface. Use this interface to get the list of preferred media types, get the current media type, and set the media type.

To get the list of preferred media types, call IMFMediaTypeHandler::GetMediaTypeByIndex. The preferred types should be taken as a hint to the client. The list might be incomplete or include partial media types. A partial media type is one that does not have all of the attributes needed to describe a valid format. For example, a partial video type might specify the color space and bit depth but not the image width or height. A partial audio type might specify the compression format and the sample rate but not the number of audio channels.

To get the stream sink's current media type, call IMFMediaTypeHandler::GetCurrentMediaType. When a stream sink is first created, it might have a default media type already set, or it might have no media type until the client sets one.

To set the media type, call IMFMediaTypeHandler::SetCurrentMediaType. Some stream sinks might not support changing the type after is has been set. Therefore, it useful to test media types before setting them. To test whether the media sink will accept a media type, (without setting the type), call IMFMediaTypeHandler::IsMediaTypeSupported.

Data Flow

Media sinks use a pull model, which means that the stream sinks request data as they need it. The client should respond in a timely manner to avoid any glitching.

Some media sinks support prerolling. Prerolling is the process of giving data to the media sink before the presentation clock starts. If a media sink supports prerolling, the media sink exposes the IMFMediaSinkPreroll interface, and the GetCharacteristics method returns the MEDIASINK_CAN_PREROLL flag. Prerolling ensures that the media sink is ready to present the first sample when the presentation clock starts. It is recommended that the client always preroll if the media sink supports it, because it can prevent glitching or gaps during playback.

Data flow to a media sink works as follows:

  1. The client sets the media types and the presentation clock. The media sink registers itself with the presentation clock to receive notifications about clock state changes.
  2. Optionally, the client queries for IMFMediaSinkPreroll. If the media sink exposes this interface, the client calls IMFMediaSinkPreroll::NotifyPreroll. Otherwise, the client skips to step 5.
  3. Each stream sink sends one or more MEStreamSinkRequestSample events. In response to each of these events, the client gets the next sample of data for that stream and calls IMFStreamSink::ProcessSample.
  4. When each stream sink receives enough preroll data, it sends an MEStreamSinkPrerolled event.
  5. The client calls IMFPresentationClock::Start to start the presentation clock.
  6. The presentation clock notifies the media sink that the clock is starting, by calling IMFClockStateSink::OnClockStart.
  7. To get more data, each stream sink sends MEStreamSinkRequestSample events. In response to each of these events, the client gets the next sample and calls ProcessSample. This step is repeated until the presentation ends.

Most media sinks process samples asynchronously, so the stream sinks can send out more than one sample request at a time.

During streaming, the client can call IMFStreamSink::PlaceMarker and IMFStreamSink::Flush at any time. Markers are described in the next section. Flushing causes the stream sink to drop any samples that it has queued but not rendered yet.

Markers

Markers provide a way for the client to indicate specific points in the stream. A marker consists of the following information:

  • The marker type, defined as a member of the MFSTREAMSINK_MARKER_TYPE enumeration.
  • Data associated with the marker. The meaning of the data depends on the marker type. Some marker types do not have data.
  • Optional data for the client's own use.

To place a marker, the client calls IMFStreamSink::PlaceMarker. The stream sink finishes processing any samples that it received before the PlaceMarker call, and then sends an MEStreamSinkMarker event.

Most media sinks keep a queue of pending samples, which they process asynchronously. Marker events must be serialized with sample processing, so the media sink should place the markers in the same queue. For example, suppose the client makes the following method calls:

  1. ProcessSample (Sample #1)
  2. ProcessSample (Sample #2)
  3. PlaceMarker (Marker #1)
  4. ProcessSample (Sample #3)
  5. PlaceMarker (Marker #2)

In this example, the stream sink must send the MEStreamSinkMarker event for marker #1 after it processes sample #2, and the event for marker #2 after it processes sample #3.

If the client flushes a stream sink, the stream sink immediately processes any markers that were in the queue. It sets the status code to E_ABORT on these events.

Some markers contain information that is relevant to the media sink:

  • MFSTREAMSINK_MARKER_TICK: Indicates there is a gap in the stream. The next sample will be a discontinuity.
  • MFSTREAMSINK_MARKER_ENDOFSEGMENT: Indicates the end of a segment or the end of a stream. The next sample (if any) might be a discontinuity.
  • MFSTREAMSINK_MARKER_EVENT: Contains an event. Depending on the event type and the implementation of the media sink, the media sink might handle the event or ignore it.

State Changes

A media sink is notified of state changes in the presentation clock through the media sink's IMFClockStateSink interface. When the client sets the presentation clock, the media sink calls IMFPresentationClock::AddClockStateSink to register itself for notifications from the clock. The following table summarizes how a media sink behaves in response to clock state changes.

Clock state change Sample processing Marker processing
OnClockStart Process samples whose time stamp is equal to or later than the clock start time. Send the MEStreamSinkMarker event when all of the samples received before the marker have been processed.
OnClockPause The media sink can fail ProcessSample while paused.
If the media sink accepts samples while paused, it must queue them until the clock restarts. Do not process any queued samples while paused.
 If there are queued samples, put markers onto the same queue. Send the marker event when the clock restarts.
Otherwise, send the marker event immediately.
OnClockRestart Process any samples that were queued while paused, and then treat the same as OnClockStart. Send MEStreamSinkMarker events for queued markers (serialized with sample processing), and then treat the same as OnClockStart.
OnClockStop Drop all queued samples. Further calls to ProcessSample can fail. Send queued marker events. On subsequent calls to PlaceMarker, send the marker event immediately.

 

In addition, stream sinks must send the following events when they have completed the state transitions:

Finalizing

Some media sinks require an extra processing step after the last sample is delivered. Typically this requirement applies to archive sinks, which must write headers or indexes into the file. If a media sink requires any final processing, it exposes the IMFFinalizableMediaSink interface.

After the client delivers the last sample, the client queries for this interface. If the media sink supports the interface, the client calls IMFFinalizableMediaSink::BeginFinalize to perform the final processing asynchronously. This method follows the standard Media Foundation asynchronous model, described in Asynchronous Callback Methods. The media sink can assume that the client will call BeginFinalize. Failure to call BeginFinalize can result in an incorrectly authored file.

Shutting Down

When the client is done using the media sink, the client calls IMFMediaSink::Shutdown. Inside this method, the media sink should break any circular reference counts. Typically, there will be circular references between the media sink and the stream sinks.

If you are using the event queue helper object to implement IMFMediaEventGenerator, call IMFMediaEventQueue::Shutdown on the event queue. This method shuts down the event queue and signals any caller that is currently waiting for an event.

After shutdown, all methods on the media sink return MF_E_SHUTDOWN, with the exception of IUnknown methods.

Media Sink Interfaces

The following table lists the standard interfaces that media sinks can expose through QueryInterface. Media sinks can also expose custom interfaces.

Interface Description
IMFMediaSink The primary interface for media sinks. (Required.)
IMFClockStateSink Used to notify the media sink when presentation clock changes state. (Required.)
IMFFinalizableMediaSink Implement if the media sink must perform a final processing step. (Optional.)
IMFGetService Implement if the media sink exposes any service interfaces. (Optional.)
IMFMediaEventGenerator Implement if the media sink sends any events. (Optional.)
IMFMediaSinkPreroll Implement if the media sink supports preroll. (Optional.)
IMFPresentationTimeSource Implement if the media sink can provide a time source for the presentation clock. (Optional.)
IMFQualityAdvise Implement if the media sink can adjust playback quality. (Optional.)

 

Optionally, a media sink can implement the following interface as a service.

Service interface Description
IMFRateSupport Reports the range of supported playback rates.

 

For more information about service interfaces and IMFGetService, see Service Interfaces.

Stream Sink Interfaces

Stream sinks must expose the following interfaces through QueryInterface.

Interface Description
IMFStreamSink The primary interface for stream sinks. (Required.)
IMFMediaEventGenerator Queues events. The IMFStreamSink interface inherits this interface. (Required.)

 

Currently no service interfaces are defined for stream sinks.

Stream Sink Events

The following table lists the events that are defined for generic stream sinks. Stream sinks can also send custom events not listed here.

Event Description
MEStreamSinkFormatChanged The stream sink's media type is no longer valid. (Optional.)
MEStreamSinkMarker A marker was processed. (Required.)
MEStreamSinkPaused The stream sink has paused. (Required.)
MEStreamSinkPrerolled Preroll is complete. (Optional.)
MEStreamSinkRateChanged The stream sink has changed playback rate. (Optional.)
MEStreamSinkRequestSample A new sample is requested. (Required.)
MEStreamSinkScrubSampleComplete A scrub request was completed. (Optional.)
MEStreamSinkStarted The stream sink has started. (Required.)
MEStreamSinkStopped The stream sink has stopped. (Required.)

 

Currently no general-purpose events are defined for media sinks. Some media sinks might send custom events.

Media Foundation Pipeline

Media Foundation Architecture