previous next

Chapter 13: Audio Services

RealSystem's Audio Services provides device-independent, cross-platform audio services to a rendering plug-in. This plug-in can use Audio Services to render audio streams without concern for the specifics of the audio hardware. Audio Services provides several useful features for rendering audio in RealSystem:

As shown in the following figure, a rendering plug-in uses the Player object to register with Audio Services and request audio data. The plug-in then writes each audio stream to a separate stream object.

Rendering Plug-in Using Audio Services

As illustrated in the next figure, Audio Services supports multiple audio streams from one or more rendering plug-ins, creating a separate stream object for each audio stream. Because Audio Services handles the output to the audio hardware, rendering plug-ins do not need to compete for access to the audio device.

Multiple Audio Streams

Interfaces

A rendering plug-in implements the following interfaces, depending on which Audio Services features it needs to use.

A rendering plug-in uses the following interface to access Audio Services functions:

Supported Input Formats

Audio Services accepts 8-bit and 16-bit Pulse Code Modulation (PCM) data. The rendering plug-in must convert audio data in other formats, such as MIDI or µ-law, to PCM before sending it to Audio Services. However, the plug-in can also write MIDI data directly to the MIDI hardware, bypassing Audio Services entirely.

The audio data can be mono or stereo at any of the following sampling rates:

When multiple rendering plug-ins send audio data to Audio Services, Audio Services mixes the inputs:

Suppose that one plug-in renders 11025 Hz stereo and another plug-in renders 22050 Hz mono at the same time. Playback performance is set to Best Audio Quality. In this case, Audio Services upsamples the 11025 Hz input to 22050 Hz and converts the mono input to stereo before mixing the two signals. It then sends a 22050 Hz stereo signal to the audio device.

Quality of Mixed Audio Streams

When multiple streams are mixed, the quality of the mixed audio output depends on the type and quality of the input streams. Some types of input mix better than others, and higher quality input always gives higher quality output.

To achieve the best possible quality, encode all mixed input streams at 44100 Hz. If the audio is digitized from an analog source, use the same audio hardware to digitize each input. Avoid mixing audio encoded at multiples of 8000 Hz with audio encoded at multiples of 11025 Hz.

Rendering Audio

Follow the steps in this section to use Audio Services to render audio data with your rendering plug-in. These steps are based on the sample rendering plug-in exaudio.cpp. "Modifying the Audio Rendering Sample Code" provides more information about this sample file.

Additional Information
Be sure to review "Chapter 6: Rendering Plug-In".

  1. In the IRMARenderer::StartStream method of the rendering plug-in, get the interface to the AudioPlayer object:
    
    QueryInterface(IID_IRMAAudioPlayer, (void**) &m_pAudioPlayer ) 
    

  2. Create an AudioStream object with IRMARenderer::OnHeader:
    
    m_pAudioPlayer-CreateAudioStream( &m_pAudioStream );
    

    If the rendering plug-in needs to render more than one audio stream, create a stream object for each stream.

  3. Determine the format of the audio data sent to the rendering plug-in. This is often in parameters passed in IRMARenderer::OnHeader. The required data are:

  4. Set the values of the audio stream parameters and then initialize the AudioStream object. The pValues parameter contains information to identify this stream uniquely.
    
    m_pAudioStream-Init( &AudioFmt, pValues); 
    

  5. In the rendering plug-in's IRMARenderer::OnPacket method, perform renderer-specific decoding or processing of each audio packet sent by RealPlayer. Set the buffer size and the start time of the packet in milliseconds. Set the uAudioStreamType member of the RMAAudioData structure to one of these values:

    Note that within a single stream you cannot write packets that have start times earlier than packets already written. If you need to process these packets first, either buffer them for later writing within the current stream or create a separate audio stream for them.

  6. Write the PCM audio data to the AudioStream object:
    
    m_pAudioStream-Write( &audioData ); 
    

    After calling IRMAAudioStream::Write for timed audio, the plug-in increments the ulAudioTime member of the RMAAudioData structure by the length of the buffer just written to get the time of the next buffer.

  7. In IRMARenderer::EndStream, which RealPlayer calls when the stream finishes, release the audio player and stream objects:
    
      if (m_pAudioPlayer)
    {
    m_pAudioPlayer-Release();
    m_pAudioPlayer = NULL;
    }
    if (m_pAudioStream)
    {
    m_pAudioStream-Release();
    m_pAudioStream = NULL;
    }

Controlling Volume

Audio Services provides the IRMAVolume interface to query, set, and mute volume, as well as register a plug-in for notifications through the IRMAVolumeAdviseSink interface. IRMAVolumeAdviseSink then lets your plug-in receive notices of changes to volume and mute settings. Audio Services enables plug-ins to control the volume of individual streams, of the final mixed stream, and of the physical audio device:

The following figure illustrates the relationships between the plug-in, RealPlayer, the audio device, and the various volume objects:

Volume Control

Implementing Midstream Playback and Instant Sounds

Audio Services lets a rendering plug-in begin playback of an audio stream at any specified time in the current presentation's timeline. It also provides a special case of midstream playback that starts a new stream at the current time. These "instant sounds" are typically linked to events such as keyboard or mouse input. The following figure shows a stream (Stream 1) being played. At time T, another audio stream starts. At the current time, an instant sound plays.

Instant Sound

The following steps and figure explain how to implement midstream playback and instant sounds:

  1. Get the AudioPlayer objects and initialize the stream as described in "Rendering Audio".

  2. Use the ulAudioTime member of the RMAAudioData structure to set the packet time to begin playing the new stream. This attribute determines the number of milliseconds into the presentation timeline to start the new stream. To start the new stream at 15 seconds into the current stream, for example, set ulAudioTime to 15000. To find out what time ulAudioTime should be set to for playback to begin with the next audio block being written, call IRMAAudioStream::Write with pData set to NULL. ulAudioTime will be set to the next audio timestamp.

    To play an instant sound, set the uAudioStreamType member of the RMAAudioData structure to INSTANTANEOUS_AUDIO. Because audio time is not significant for instantaneous playback, ulAudioTime is set to 0. Audio Services plays the instant sound immediately, mixing it with any stream currently playing, whether a timed track or another instantaneous sound.

  3. Call IRMAAudioStream::Write.

  4. Send additional packets of the new stream using IRMAAudioStream::Write. Increment ulAudioTime by the length in milliseconds of the last buffer sent.
Mid-Stream Playback and Instant Sound

Using Post-Processed Audio Data

An application that needs access to data sent to the audio device, such as an application that adds sound effects to a stream, can receive pre-mix audio data (individual decoded streams) or post-mix audio data (final mixed stream). The plug-in receives data as headerless buffers that it can modify and return to Audio Services, or even pass to another plug-in. Various Audio Services methods let the plug-in obtain the stream's sampling rate, number of channels, and audio format attributes. As explained in "Supported Input Formats", the audio output format depends on the inputs to Audio Services.

Note
Your plug-in must be able to receive and modify post-processed audio data synchronously in real-time. Ensure that your plug-in platform is capable of performing such real-time processing.

Getting Pre-Mix Audio Data

As illustrated in the following figure, a plug-in can examine and modify pre-mix audio data, which is the decoded data from a single stream, before Audio Services mixes it with other streams.

Pre-Mix Audio Data

The following steps explain how to get pre-mix audio data. An example plug-in that uses the pre-mix interface is premixrd.cpp. See "Modifying the Audio Rendering Sample Code" for more information on using this code.

  1. Implement an IRMAAudioStreamInfoResponse class.

  2. Implement an IRMAAudioHook class.

  3. Do the following in your IRMARenderer::StartStream method:

    1. Create an IRMAAudioPlayer using IUnknown::QueryInterface.

    2. Create an IRMAAudioHook interface.

    3. Create an IRMAAudioStreamInfoResponse interface.

    4. Register your IRMAAudioStreamInfoResponse interface with the AudioPlayer:
      
      m_pAudioPlayer-SetStreamInfoResponse( m_pResp ); 
      

  4. When the AudioPlayer object passes the stream (or streams) to your renderer with IRMAAudioStreamInfoResponse::OnStream, test the appropriate IRMAValues name/value pair to determine if this is the desired stream. For example, the following sample code locates the stream with "MimeType" equal to audio/x-pn-wav:
    
    {
    IRMAValues* pValues = 0;
    IRMABuffer* pMimeType = 0;

    pValues = pAudioStream-GetStreamInfo();
    pValues-GetPropertyCString("MimeType", pMimeType);

    char* pMime = (char*) pMimeType-GetBuffer();
    char* pStreamName = (char*) m_pHookStreamName-GetBuffer();

    /* In this example, let's hook all wav streams. */
    if (pMime && pStreamName && (!strcmp(pMime, pStreamName)))
    {
    /* Add pre mix hook on this stream. */
    pAudioStream-AddPreMixHook(m_pHook, FALSE);
    }

    return PNR_OK;
    }

    The call to pAudioStream-AddPreMixHook(m_pHook, FALSE) adds the pre-mix hook. The m_pHook parameter is the pointer to the IRMAAudioHook interface. The bDisableWrite parameter is set to FALSE to send the stream to the audio mixer. Set bDisableWrite to TRUE to keep the stream out of audio mixing. Remove a hook with pAudioStream-RemovePreMixHook(m_pHook).

  5. After IRMAAudioStream::AddPreMixHook is called, the AudioStream object calls IRMAAudioHook::OnInit and passes the audio format of the audio stream. Within this method, initialize the plug-in as needed.

  6. For each buffer of audio data in the stream, the AudioStream object calls IRMAAudioHook::OnBuffer and passes the audio data from the stream. Copy the contents of the buffer and process as needed. Do the following if you need to modify the audio data:

    1. Create an IRMABuffer interface to store the modified audio data.

    2. Modify the data as needed.

    3. Return a pointer to the modified data as the second parameter of IRMAAudioHook::OnBuffer.

Getting Post-Mix Audio Data

As shown in the figure below, a plug-in can modify the post-mix audio data, which is the final audio stream after all audio streams are mixed.

Post-Mix Audio Data

Complete the following steps to get post-mix audio data. An example plug-in that uses the post-mix interface is included in pstmixrd.cpp. See "Modifying the Audio Rendering Sample Code" for more information on using this code.

  1. Implement an IRMAAudioHook class.

  2. Do the following in the IRMARenderer::StartStream method:

    1. Get and save an IRMAAudioPlayer interface through IUnknown::QueryInterface.

    2. Create an IRMAAudioHook interface.

  3. With IRMARenderer::OnHeader, add the post-mix hook:
    
    // Add post process hook

    BOOL bDisableWrite = FALSE; //write data to the audio device

    BOOL bFinal = FALSE;

    m_pAudioPlayer-AddPostMixHook(m_pHook, bDisableWrite, bFinal);

    Specifying bDisableWrite as TRUE prevents Audio Services from sending audio data to the audio device. The plug-in then must write the data to the audio device itself. Even when the plug-in writes the data itself, Audio Services provides all renderers with time synchronization based on a real-time clock. Remove a hook with M_pAudioPlayer-RemovePostMixHook(m_pHook).

  4. The AudioPlayer object calls IRMAAudioHook::OnInit with the audio format of the hooked data.

  5. The AudioSession object calls IRMAAudioHook::OnBuffer with the post-mixed audio data.

  6. IRMAAudioHook::OnBuffer may change the data; but it must create its own IRMABuffer to do this (use IRMACommonClassFactory) and return the modified data in the pAudioOutData parameter.

    Note
    If there are multiple audio players due to multiple timelines,the post-mix hook gets data only for its audio player. This data may differ from what gets written to the audio device if there are other audio streams in other timelines as well.

Receiving Notification of a Dry Stream

RealSystem provides IRMAStream::ReportRebufferStatus as a standard means for a plug-in to notify RealPlayer that the available data has dropped to a critically low level and rebuffering should occur. If your renderer does not send buffered data because, for example, the rendered data stems from interactive input, you can implement IRMADryNotification to receive notification of a stream running dry, which occurs when the player must write data to the audio device but it does not have enough data to write.

Additional Information
See "Using the Stream Object".

Set up a notification response object with IRMAAudioStream::AddDryNotification. The player core then uses IRMADryNotification::OnDryNotification to notify your renderer of a stream running dry. This method passes two parameters:

The renderer must take action synchronously within the function call. It is acceptable for the renderer not to respond. It just means that silence occurs until the renderer delivers the next packets.

Synchronizing to Audio

RealSystem synchronizes playback to a presentation's audio track. If there is no audio track, it synchronizes playback based on the system time.

During start-up, rendering plug-ins request periodic time synchronization callbacks. The audio hardware generates the synchronization signals based on the actual playback of the audio track. The AudioDevice object passes these signals back to the client, which then issues callbacks to the rendering plug-in through IRMARenderer::OnTimeSync.

The rendering plug-in's IRMARenderer::GetRendererInfo method specifies the granularity of the time synchronization that the plug-in needs. The player issues callbacks as closely as possible to the requested interval. The minimum granularity is 20 milliseconds.

Modifying the Audio Rendering Sample Code

The RealSystem SDK includes sample Audio Services plug-ins that you can use as a starting point for creating your own plug-in. The /samples directory contains code for several basic Audio Services plug-ins. The /advanced subdirectory under /samples contains plug-in samples that use more advanced Audio Services features:

Do the following to modify the code in any of the sample files:

  1. Back up the original sample files.

  2. Modify the MIME type of data rendered or captured by the plug-in.

  3. Write your application-specific code as needed.

  4. Build and test your plug-in.

    Additional Information
    See "Compiling a Plug-In".


Copyright © 2000 RealNetworks
For technical support, please contact supportsdk@real.com.
This file last updated on 05/17/00 at 12:50:21.
previous next