Compartilhar via

videoEffects module

  • Reference
Package:
@microsoft/teams-js

Note

This namespace is in Beta and provided as a preview for developers and may change based on feedback that we receive. Do not use this API in a production environment.

Module to support video extensibility of the SDK

Interfaces

VideoBufferData

Represents a video frame
VideoFrame

VideoFrame definition, align with the W3C spec: https://www.w3.org/TR/webcodecs/#videoframe-interface. The current version of typescript doesn't have the definition of VideoFrame so we have to define it here. At runtime it can be cast to VideoFrame directly: (videoFrame as VideoFrame).
VideoFrameConfig

Video frame configuration supplied to the host to customize the generated video frame parameters, like format

Type Aliases

notifyErrorFunctionType

Notify error function type
notifyVideoFrameProcessedFunctionType

Notify video frame processed function type
RegisterForVideoFrameParameters

Callbacks and configuration supplied to the host to process the video frames.
VideoBufferHandler

Video frame call back function definition The callback will be called on every frame when running on the supported host. We require the frame rate of the video to be at least 22fps for 720p, thus the callback should process a frame timely. The video app should call notifyVideoFrameProcessed to notify a successfully processed video frame. The video app should call notifyError to notify a failure. When the failures accumulate to a certain number, the host will see the app is "frozen" and ask the user to close it or not.
VideoEffectCallback

Video effect change call back function definition Return a Promise which will be resolved when the effect is prepared, or throw an EffectFailureReason on error.
VideoFrameData

Video frame data extracted from the media stream. More properties may be added in the future.
VideoFrameHandler

Video frame call back function definition. The callback will be called on every frame when running on the supported host. We require the frame rate of the video to be at least 22fps for 720p, thus the callback should process a frame timely. The video app should resolve the promise to notify a successfully processed video frame. The video app should reject the promise to notify a failure. When the failures accumulate to a certain number, the host will see the app is "frozen" and ask the user to close it or not.

Enums

EffectChangeType

Video effect change type enum
EffectFailureReason

Predefined failure reasons for preparing the selected video effect
VideoFrameFormat

Video frame format enum, currently only support NV12

Functions

isSupported()

Checks if video capability is supported by the host.
notifySelectedVideoEffectChanged(EffectChangeType, undefined | string)

Video extension should call this to notify host that the current selected effect parameter changed. If it's pre-meeting, host will call videoEffectCallback immediately then use the videoEffect. If it's the in-meeting scenario, we will call videoEffectCallback when apply button clicked.
registerForVideoEffect(VideoEffectCallback)

Register a callback to be notified when a new video effect is applied.
registerForVideoFrame(RegisterForVideoFrameParameters)

Register callbacks to process the video frames if the host supports it.

Example

videoEffects.registerForVideoFrame({
  videoFrameHandler: async (videoFrameData) => {
    const originalFrame = videoFrameData.videoFrame as VideoFrame;
    try {
      const processedFrame = await processFrame(originalFrame);
      return processedFrame;
    } catch (e) {
      throw e;
    }
  },
  videoBufferHandler: (
    bufferData: VideoBufferData,
    notifyVideoFrameProcessed: notifyVideoFrameProcessedFunctionType,
    notifyError: notifyErrorFunctionType
    ) => {
      try {
        processFrameInplace(bufferData);
        notifyVideoFrameProcessed();
      } catch (e) {
        notifyError(e);
      }
    },
  config: {
    format: videoEffects.VideoPixelFormat.NV12,
  }
});

Function Details

isSupported()

Note

This API is in Beta and provided as a preview for developers and may change based on feedback that we receive. Do not use this API in a production environment.

Checks if video capability is supported by the host.

function isSupported(): boolean

Returns

boolean

boolean to represent whether the video capability is supported

notifySelectedVideoEffectChanged(EffectChangeType, undefined | string)

Note

This API is in Beta and provided as a preview for developers and may change based on feedback that we receive. Do not use this API in a production environment.

Video extension should call this to notify host that the current selected effect parameter changed. If it's pre-meeting, host will call videoEffectCallback immediately then use the videoEffect. If it's the in-meeting scenario, we will call videoEffectCallback when apply button clicked.

function notifySelectedVideoEffectChanged(effectChangeType: EffectChangeType, effectId: undefined | string)

Parameters

effectChangeType
EffectChangeType

the effect change type.

effectId

undefined | string

Newly selected effect id.

registerForVideoEffect(VideoEffectCallback)

Note

This API is in Beta and provided as a preview for developers and may change based on feedback that we receive. Do not use this API in a production environment.

Register a callback to be notified when a new video effect is applied.

function registerForVideoEffect(callback: VideoEffectCallback)

Parameters

callback
VideoEffectCallback

Function to be called when new video effect is applied.

registerForVideoFrame(RegisterForVideoFrameParameters)

Note

This API is in Beta and provided as a preview for developers and may change based on feedback that we receive. Do not use this API in a production environment.

Register callbacks to process the video frames if the host supports it.

Example

videoEffects.registerForVideoFrame({
  videoFrameHandler: async (videoFrameData) => {
    const originalFrame = videoFrameData.videoFrame as VideoFrame;
    try {
      const processedFrame = await processFrame(originalFrame);
      return processedFrame;
    } catch (e) {
      throw e;
    }
  },
  videoBufferHandler: (
    bufferData: VideoBufferData,
    notifyVideoFrameProcessed: notifyVideoFrameProcessedFunctionType,
    notifyError: notifyErrorFunctionType
    ) => {
      try {
        processFrameInplace(bufferData);
        notifyVideoFrameProcessed();
      } catch (e) {
        notifyError(e);
      }
    },
  config: {
    format: videoEffects.VideoPixelFormat.NV12,
  }
});

function registerForVideoFrame(parameters: RegisterForVideoFrameParameters)

Parameters

parameters
RegisterForVideoFrameParameters

Callbacks and configuration to process the video frames. A host may support either VideoFrameHandler or VideoBufferHandler, but not both. To ensure the video effect works on all supported hosts, the video app must provide both VideoFrameHandler and VideoBufferHandler. The host will choose the appropriate callback based on the host's capability.