Acquiring Image and Audio Data on the Phone

On this page:	Download:
You Will Learn | Overview of the Solution - Capturing Image Data, Recording Audio Data | Inside the Implementation - Capturing Image Data, Using XNA Interop to Record Audio | Logging Errors and Diagnostic Information on the Phone

You Will Learn

• How to capture and store image data from the Windows Phone camera.
• How to capture and store audio data from the Windows Phone microphone.

The Tailspin Surveys mobile client application allows users to capture images from the device's camera and record audio from the device's microphone as answers to survey questions. The application saves the captured data as part of the survey answer, and this data is sent to the Tailspin Surveys web service when the user synchronizes the mobile client application.

The mobile client application can capture image and audio data.

Overview of the Solution

The techniques you use to capture audio and image data are different. To capture image data from the camera, you use a "chooser," and to capture audio data from the microphone, you must use interop with the XNA® development platform framework on the phone.

Capturing Image Data

The chooser for capturing image data is the CameraCaptureTask. When you use a chooser, the operating system deactivates your application and runs the chooser as a new process. When the chooser has completed its task, the operating system reactivates your application and delivers any data to your application using a callback method. The developers at Tailspin chose to implement this using the Observable class from the Rx library on the phone. The Tailspin Surveys mobile client application saves the captured picture to isolated storage along with the other survey answers and also displays the picture in the view.

Recording Audio Data

To access the microphone on the Windows Phone device, you have to use XNA; it's not possible to access it directly from Silverlight. To interoperate with XNA, you must use an XNA asynchronous event dispatcher to connect to the XNA events from Silverlight. Your application can then handle the microphone events that deliver the raw audio data to your application. Your application must then convert or encode the audio data to a valid sound format before saving it in isolated storage.

Inside the Implementation

Now is a good time to take a more detailed look at the code that captures image data and records audio.

As you go through this section, you may want to download the Windows Phone Tailspin Surveys application from the Microsoft Download Center.

Capturing Image Data

The ICameraCaptureTask interface in the TailSpin.PhoneClient.Adapters project defines signatures for a property, an event, and a method. The following code example shows this interface.

public interface ICameraCaptureTask
{
  SettablePhotoResult TaskEventArgs
  {
      get;
      set;
  }

  event EventHandler<SettablePhotoResult> Completed;

  void Show();
}

The ICameraCaptureTask interface is implemented by the CameraCaptureTaskAdapter class, which adapts the CameraCaptureTask class from the API. The purpose of adapting the CameraCaptureTask class with a class that implements ICameraCaptureTask is to create a loosely coupled class that is testable. The following code example shows the class.

public class CameraCaptureTaskAdapter : ICameraCaptureTask
{
  public CameraCaptureTaskAdapter()
  {
      WrappedSubject = new CameraCaptureTask();
      ...
  }

  private CameraCaptureTask WrappedSubject { get; set; }

  public SettablePhotoResult TaskEventArgs
  {
    get
    {
      return new SettablePhotoResult(WrappedSubject.TaskEventArgs);
    }
    set
    {
      WrappedSubject.TaskEventArgs = value;
    }
  }

  public event EventHandler<SettablePhotoResult> Completed;

  ...

  public void Show()
  {
    WrappedSubject.Show();
  }

  ...
}

The SettablePhotoResult class provides an implementation of the PhotoResult class where the ChosenPhoto and OriginalFileName properties are settable. The following code example shows the class.

public class SettablePhotoResult : PhotoResult
{
  public SettablePhotoResult(PhotoResult photoResult)
  {
    ChosenPhoto = photoResult.ChosenPhoto;
    OriginalFileName = photoResult.OriginalFileName;
    Error = photoResult.Error;
  }

  public SettablePhotoResult()
  {
  }

  public new Stream ChosenPhoto { get; set; }
  public new string OriginalFileName { get; set; }
  public new Exception Error { get; set; }
}

When TakeSurveyViewModel creates an instance of the PictureQuestionViewModel class, it passes in a new instance of the CameraCaptureTaskAdapter class, which in turn creates an instance of the CameraCaptureTask class.

The following code example shows how the CameraCaptureCommand delegate command is defined in the constructor for the PictureQuestionViewModel class. This command uses the Capturing property to check whether the application is already in the process of capturing a picture and to control whether the command can be invoked from the UI. The method displays a picture if there is already one saved for this question.

Markus Says:
	You can use the Exchangeable Image File (EXIF) data in the picture to determine the correct orientation for displaying the picture.

The method then uses the CameraCaptureTaskAdapter instance, which will launch the chooser for taking the photograph and return the captured picture.

private readonly ICameraCaptureTask task;
...

public PictureQuestionViewModel(QuestionAnswer questionAnswer, 
  ICameraCaptureTask cameraCaptureTask, IMessageBox messageBox)
  : base(questionAnswer)
{
  this.CameraCaptureCommand =
    new DelegateCommand(this.CapturePicture, () => !this.Capturing);
  if (questionAnswer.Value != null)
  {
    this.LoadPictureBitmap(questionAnswer.Value);
  }
  this.task = cameraCaptureTask;
  ...
}

public DelegateCommand CameraCaptureCommand { get; set; }
...

public bool Capturing
{
  get { return this.capturing; }
  set
  {
    if (this.capturing != value)
    {
      this.capturing = value;
      this.RaisePropertyChanged(() => this.Capturing);
    }
  }
}

...

private void CapturePicture()
{
  if (!this.Capturing)
  {
    this.task.Show();
    this.Capturing = true;
    this.CameraCaptureCommand.RaiseCanExecuteChanged();
  }
}

The following code examples show how the constructor uses the Observable.FromEvent method to specify how to handle the Completed event raised by the CameraCaptureTask chooser object when the user has finished with the chooser. The first example shows how the application saves the picture if the CameraCaptureTask succeeds.

Observable.FromEvent<SettablePhotoResult>(
  h => this.task.Completed += h, 
  h => this.task.Completed -= h)
  .Where(e => e.EventArgs.ChosenPhoto != null)
  .Subscribe(result =>
  {
    this.Capturing = false;
    SavePictureFile(result.EventArgs.ChosenPhoto);
    this.Answer.Value = this.fileName;
    this.RaisePropertyChanged(string.Empty);
    this.CameraCaptureCommand.RaiseCanExecuteChanged();
  });

The second example shows how the CameraCaptureCommand command is re-enabled if the user didn't take a picture.

Observable.FromEvent<SettablePhotoResult>(
  h => this.task.Completed += h,
  h => this.task.Completed -= h)
  .Where(e => e.EventArgs.ChosenPhoto == null && 
    e.EventArgs.Error == null)
  .Subscribe(p =>
  {
    this.Capturing = false;
    this.CameraCaptureCommand.RaiseCanExecuteChanged();
  });

The third example shows how a message box containing an error message is displayed to the user, if the CameraCaptureTask fails.

Observable.FromEvent<SettablePhotoResult>(
  h => this.task.Completed += h, 
  h => this.task.Completed -= h)
  .Where(e => e.EventArgs.Error != null && 
    !string.IsNullOrEmpty(e.EventArgs.Error.Message))
  .Subscribe(p =>
  {
    this.Capturing = false;
    this.messageBox.Show(p.EventArgs.Error.Message);
    this.CameraCaptureCommand.RaiseCanExecuteChanged();
  });

Markus Says:
	Using Rx means we can filter on just the events and event parameter values that we're interested in by using simple LINQ expressions—all in compact and easy-to-read code.

The SavePictureFile method uses an efficient approach to persisting the captured image to isolated storage, which avoids writing a single byte of the image to the file during each iteration of a loop. This has the additional advantage of not requiring a progress indicator to inform the user of progress through the save process. The following code example shows the method.

private void SavePictureFile(Stream chosenPhoto)
{
  SavingPictureFile = true;

  // Store the image bytes.
  byte[] imageBytes = new byte[chosenPhoto.Length];
  chosenPhoto.Read(imageBytes, 0, imageBytes.Length);

  // Seek back so we can create an image.
  chosenPhoto.Seek(0, SeekOrigin.Begin);

  // Create an image from the stream.
  var imageSource = PictureDecoder.DecodeJpeg(chosenPhoto);
  this.Picture = imageSource;

  // Save the stream
  var isoFile = IsolatedStorageFile.GetUserStoreForApplication();
  using (var stream = isoFile.CreateFile(filename))
  {
    stream.Write(imageBytes, 0, imageBytes.Length);
  }

  SavingPictureFile = false;
}

The drawback with this method is that the CameraCaptureTask class returns a high resolution image, which the SavePictureFile method simply writes to a file. The PictureQuestionViewModel also displays the captured image. However, since high-resolution images consume a lot of memory, the mobile application client crashes when a survey contains too many picture questions containing captured images. An alternative approach would be to use the CameraCaptureTask class for image capture, and then reduce the resolution of the captured image before consuming it. Another alternative approach would to be to use the PhotoCamera API class for image capture, and configure the capture resolution prior to capturing images.

Jana Says:
	If your application is camera intensive, you should consider using the PhotoCamera API class, which allows you to configure functionality such as image capture, focus, resolution, and flash mode.

Using XNA Interop to Record Audio

Before the Tailspin mobile client application can handle events raised by XNA objects, such as a Microphone object, it must create an XNA asynchronous dispatcher service. The following code example from the VoiceQuestionViewModel class shows how this is done.

...
private XnaAsyncDispatcher xnaAsyncDispatcher;
...

public VoiceQuestionViewModel(QuestionAnswer questionAnswer, 
  IIsolatedStorageFacade isolatedStorageFacade,
  INavigationService navigationService)
    : base(questionAnswer)
{
  ...
  xnaAsyncDispatcher = new XnaAsyncDispatcher(TimeSpan.FromMilliseconds(50)));
  xnaAsyncDispatcher.StartService();
  ...
}

The VoiceQuestionView.xaml file defines two buttons, one toggles recording on and off, and the other plays back any saved audio. The recording toggle button is bound to the DefaultActionCommand command in the view model, and the play button is bound to the PlayCommand command in the view model.

The DefaultAction command uses the StartRecording and StopRecording methods in the VoiceQuestionViewModel class to start and stop audio recording. The following code example shows the StartRecording method.

private MediaState priorMediaState;
...

private void StartRecording()
{
  StopPlayback();

  priorMediaState = MediaPlayer.State;
  if (priorMediaState == MediaState.Playing)
  {
    MediaPlayer.Pause();
  }

  var mic = Microphone.Default;
  if (mic.State == MicrophoneState.Started)
  {
    mic.Stop();
  }

  this.stream = new MemoryStream();

  buffer = new byte[mic.GetSampleSizeInBytes(mic.BufferDuration)]; 

  this.observableMic = Observable.FromEvent<EventArgs>(
    h => mic.BufferReady += h, h => mic.BufferReady -= h)
    .Subscribe(p => this.CaptureMicrophoneBufferResults());
  mic.Start();
}

This method determines if a media item is being played, and if it is, the media item is paused. The method then gets a reference to the default microphone on the device and creates a MemoryStream instance to store the raw audio data.

Note

You can find the Microphone class in the Microsoft.Xna.Framework.Audio namespace.

Markus Says:
	The Windows Phone API does not include any methods to convert audio formats. You can find the WaveFormatter class in the TailsSpin.PhoneClient.Infrastructure namespace.

The method uses the Observable.FromEvent method to subscribe to the microphone's BufferReady event, and whenever the event is raised, the application calls CaptureMicrophoneBufferResults to capture the raw audio data. Finally, the method starts the microphone.

The following code example shows the StopRecording method. It creates a WaveFormatter instance to convert the raw audio data to the WAV format before using the instance to write the audio data to isolated storage. It then disposes of the MemoryStream, Microphone, and WaveFormatter instances and attaches the name of the saved audio files to the question. Finally, if a media item was playing and was paused when the StartRecording method was called, the method resumes playback of the media item.

private MediaState priorMediaState;
...

private void StopRecording()
{
  var mic = Microphone.Default;
  this.CaptureMicrophoneBufferResults();
  mic.Stop();

  this.formatter = new WaveFormatter(this.wavFileName, 
    (ushort)mic.SampleRate, 16, 1, isolatedStorageFacade);
  this.formatter.WriteDataChunk(stream.ToArray());
           
  this.stream.Dispose();
  this.observableMic.Dispose();
  this.formatter.Dispose();
  this.formatter = null;
  this.buffer = null;
  this.Answer.Value = this.wavFileName;

  if (priorMediaState == MediaState.Playing)
  {
    MediaPlayer.Resume();
  }
}

The play button in the VoiceQuestionView view plays the recorded audio by using the SoundEffect class from the Microsoft.Xna.Framework.Audio namespace. The following code example shows the Play method from the VoiceQuestionViewModel class that loads audio data from isolated storage and plays it back. Before loading and playing audio data from isolated storage, the method determines if a media item is already being played, and if it is, the media item is paused. Once playback of the audio data has finished, the method resumes playback of the original media item.

private MediaState priorMediaState;
...

private void Play()
{
  priorMediaState = MediaPlayer.State;
  if (priorMediaState == MediaState.Playing)
  {
    MediaPlayer.Pause();
  }

  fileSystem = IsolatedStorageFile.GetUserStoreForApplication();

  fileStream = fileSystem.OpenFile(this.wavFileName, FileMode.Open, 
    FileAccess.Read);

  try
  {
    soundEffect = SoundEffect.FromStream(fileStream);
    soundEffectInstance = soundEffect.CreateInstance();
    soundEffectInstance.Play();
  }
  catch (ArgumentException)
  {
  }

  if (priorMediaState == MediaState.Playing)
  {
    MediaPlayer.Resume();
  }
}

Logging Errors and Diagnostic Information on the Phone

The sample application does not log any diagnostic information or details of error conditions from the Tailspin mobile client application. Tailspin plans to add this functionality to a future version of the mobile client application after they evaluate a number of tradeoffs.

For example, Tailspin must decide whether to keep a running log on the phone or simply report errors to a service as and when they occur. Keeping a log on the phone will use storage, so Tailspin would have to implement a mechanism to manage the amount of isolated storage used for log data, perhaps by keeping rolling logs of recent activity or implementing a purge policy. Sending error data as it occurs minimizes the storage requirements, but it makes it harder to access data about the state of the application before the error occurred. Tailspin would also need to develop a robust way to send the error information to a service, while transferring log files could take place during the application's standard synchronization process. Collecting logs also makes it easier to correlate activities on the phone with activities in the various Tailspin Surveys services.

Tailspin must also consider the user experience. A privacy policy would require a user to opt into collecting diagnostic information, and Tailspin might want to include options that would enable users to set the level of logging detail to collect.

Next Topic | Previous Topic | Home

Last built: May 25, 2012