MedaiSync是android M新加入的API，可以幫助應用視音訊的同步播放，如同官網介紹的

From Andriod M:
MediaSync:
class which helps applications to synchronously render audio and video streams. The audio buffers are submitted in non-blocking fashion and are returned via a callback. It also supports dynamic playback rate.

MediaSync的基本用法

第一步：初始化MediaSync, 初始化mediaCodec和AudioTrack, 將AudioTrack和surface傳給MeidaSync

MediaSync sync = new MediaSync();
sync.setSurface(surface);
Surface inputSurface = sync.createInputSurface();
...
// MediaCodec videoDecoder = ...;
videoDecoder.configure(format, inputSurface, ...);
...
sync.setAudioTrack(audioTrack);
 

第二步： MediaSync只會對audiobuffer做操作，一個是代表寫入的queueAudio方法，一個是代表寫完了的回撥方法，也就是下面的

onAudioBufferConsumed
sync.setCallback(new MediaSync.Callback() {
    @Override
    public void onAudioBufferConsumed(MediaSync sync, ByteBuffer audioBuffer, int bufferId) {
        ...
    }
}, null);

第三步：設定播放速度

// This needs to
 
 be done since sync is paused on creation.
sync.setPlaybackParams(new PlaybackParams().setSpeed(1.f));

第四步：開始流轉音視訊buffer，這裡就和MediaCodec的基本呼叫流程一樣了，當拿到audioBuffer後，通過queueAudio將buffer給MediaSync，在對應的回撥方法中release播放出去，至於video部分，直接releaseOutputBuffer即可

for (;;) {
  ...
  // send video frames to surface for rendering, e.g., call
  videoDecoder.releaseOutputBuffer(videoOutputBufferIx,videoPresentationTimeNs);
  ...
  sync.queueAudio(audioByteBuffer, bufferId, audioPresentationTimeUs); // non-blocking.
  // The audioByteBuffer and bufferId will be returned via callback.
}

第五步：播放完畢

 sync.setPlaybackParams(new PlaybackParams().setSpeed(0.f));
 sync.release();
 sync = null;

如果用的是MediaCodec的非同步流程，如下，通過下面的程式碼可以更好的理解video buffer和audio buffer分別是怎麼處理的

onOutputBufferAvailable(MediaCodec codec, int bufferId, BufferInfo info) {
    // ...
    if (codec == videoDecoder) {
        codec.releaseOutputBuffer(bufferId, 1000 * info.presentationTime);
    } else {
        ByteBuffer audioByteBuffer = codec.getOutputBuffer(bufferId);
        sync.queueAudio(audioByteBuffer, bufferId, info.presentationTime);
    }
    // ...
}
onAudioBufferConsumed(MediaSync sync, ByteBuffer buffer, int bufferId) {
    // ...
    audioDecoder.releaseBuffer(bufferId, false);
    // ...
}

MediaSync的關鍵變數與方法

SyncParams：Android M新加入的API，用於控制AV同步的方法
1、倍速播放時如何處理audio

int AUDIO_ADJUST_MODE_DEFAULT
System will determine best handling of audio for playback rate adjustments.
Used by default. This will make audio play faster or slower as required by the sync source without changing its pitch; however, system may fall back to some other method (e.g. change the pitch, or mute the audio) if time stretching is no longer supported for the playback rate.

int AUDIO_ADJUST_MODE_RESAMPLE
Resample audio when playback rate must be adjusted.
This will make audio play faster or slower as required by the sync source by changing its pitch (making it lower to play slower, and higher to play faster.)

int AUDIO_ADJUST_MODE_STRETCH
Time stretch audio when playback rate must be adjusted.
This will make audio play faster or slower as required by the sync source without changing its pitch, as long as it is supported for the playback rate.

2、選擇avsync的基準

int SYNC_SOURCE_AUDIO
Use audio track for sync source. This requires audio data and an audio track.

int SYNC_SOURCE_DEFAULT
Use the default sync source (default). If media has video, the sync renders to a surface that directly renders to a display, and tolerance is non zero (e.g. not less than 0.001) vsync source is used for clock source. Otherwise, if media has audio, audio track is used. Finally, if media has no audio, system clock is used.

int SYNC_SOURCE_SYSTEM_CLOCK
Use system monotonic clock for sync source.

int SYNC_SOURCE_VSYNC
Use vsync as the sync source. This requires video data and an output surface that directly renders to the display, e.g. SurfaceView

PlaybackParams Android M 新加入的API，主要用於控制倍速播放

get & setPlaybackParams (PlaybackParams params)
Gets and Sets the playback rate using PlaybackParams.

MediaTimestamp Android M新加入的API

MediaTimestamp getTimestamp ()
Get current playback position.

MediaSyncExample

瞭解了MediaSync的基本用法和關鍵變數之後，我們可以參考cts中的程式碼寫一個demo，基於這個demo測試MediaSync的音視訊同步結果，研究背後的原理。
DEMO地址：https://github.com/zhanghuicuc/MediaSyncExample

DEMO對應的時序圖如下

關鍵的同步部分是在queueAudio之後開始進行的，需要注意的是，這裡我們看到了熟悉的MediaClock和VideoFrameScheduler，它們都在libstagefright中，並且在NuPlayer的同步中就用到了它們。

MediaSync的同步邏輯

下面我會先簡要的介紹MediaSync avsync邏輯中的關鍵點，最後再進行詳細的程式碼分析。

Video部分

1、對實際送顯時間的計算

void MediaSync::onDrainVideo_l()
         int64_t nowUs = ALooper::GetNowUs();
        BufferItem *bufferItem = &*mBufferItems.begin();
        int64_t itemMediaUs = bufferItem->mTimestamp / 1000;
       //這裡就是呼叫MediaClock的getRealTimeFor方法，得到“視訊幀應該顯示的時間”
        int64_t itemRealUs = getRealTime(itemMediaUs, nowUs);

2、利用vsync訊號調整itemRealUs

void MediaSync::onDrainVideo_l()
        itemRealUs = mFrameScheduler->schedule(itemRealUs * 1000) / 1000;
       //計算兩倍vsyncDuration
        int64_t twoVsyncsUs = 2 * (mFrameScheduler->getVsyncPeriod() / 1000);

3、提前2倍vsync duration時間送顯

void MediaSync::onDrainVideo_l()

if (itemRealUs <= nowUs + twoVsyncsUs) {
            //如果當前時間距離itemRealUs已經不足2*vsyncDuration，則趕緊去顯示
            //將buffer的timeStamp設定為修正後的itemRealUs
            bufferItem->mTimestamp = itemRealUs * 1000;
            bufferItem->mIsAutoTimestamp = false;
            //queuebuffer給output，送去SF顯示
            renderOneBufferItem_l(*bufferItem);
} else {
                sp<AMessage> msg = new AMessage(kWhatDrainVideo, this);
                msg->post(itemRealUs - nowUs - twoVsyncsUs);
                mNextBufferItemMediaUs = itemMediaUs;
}

前面我們分析NuPlayer的avsync時，曾經發現了一個問題，即NuPlayer實際送給render的video buffer timestamp並不是經過frameScheduler調整後的時間戳，看來在MediaSync中修正了這個問題

Audio部分

1、current play time 的計算

status_t MediaSync::updateQueuedAudioData

    int64_t nowUs = ALooper::GetNowUs();
    int64_t nowMediaUs = presentationTimeUs
            - (getDurationIfPlayedAtNativeSampleRate_l(mNumFramesWritten)
            - getPlayedOutAudioDurationMedia_l(nowUs));
    mMediaClock->updateAnchor(nowMediaUs, nowUs, maxMediaTimeUs);

至於其中關鍵的getPlayedOutAudioDurationMedia_l方法，也和NuPlayer中的完全一樣

小結

Video部分：
和NuPlayer的相似度在95%以上，
不同點在於真正利用了vsync調整後的時間戳作為顯示時間戳

Audio部分：
完全就是NuPlayer中的邏輯！

MediaSync的架構

先來介紹一下MediaSync的基本架構，codec, surface, surfaceFlinger之間的關係如下

我們知道framework中各個模組都是各司其職的，MediaCodec專門做解碼的工作，Surface & SurfaceFlinger專門做畫面的組合、繪製的工作，現在要專門搞一個負責AVSync的模組MediaSync，又要獨立於MediaCodec和Suface，要怎麼搞呢？

回憶MediaSync的基本呼叫流程，在第一步中我們進行了如下了的工作

MediaSync sync = new MediaSync();
sync.setSurface(surface); 
這裡的第一個surface是應用的surfaceHolder.getSurface()拿到的Surface

Surface inputSurface = sync.createInputSurface();
這裡的第二個Surface是MediaSync返回給我們的surface，
是一個新的Surface，已經和第一個surface不同了
...
// MediaCodec videoDecoder = ...;
videoDecoder.configure(format, inputSurface, ...);
在MediaCodec.configure時傳入的是MediaSync返回的Surface，而不是應用本身的surface了

我們之前在ExoPlayer和NuPlayer中看到的音視訊同步策略都是在codec和Surface之間來做的。現在想要加一個可以配合MediaCodec使用的MediaSync模組，則需要做成下面的樣子

圖中Surface2是應用本身的Surface，定義為MediaSync的output，它是一個GraphicBufferProducer；Surface1是MediaSync提供給MediaCodec configure用的，定義為MediaSync的input，它是一個GraphicBufferConsumer
在MediaSync的音視訊同步控制就是以audio clock為基準，控制Surface1和Surface2之間的視訊buffer

buffer流轉的順序如下：
1、surface1拿到codec送來的buffer
acquire buffer from Input
2、在MediaSync中對buffer的timestamp進行調整後，把buffer給surface2
queue buffer to Output
3、surface2將這個buffer給surfaceFlinger顯示
detached buffer from output
4、和mediacodec一樣，再把buffer給input
release buffer to Input

先來看一下這兩個surface是怎麼來的
1、應用本身的buffer，即圖中的surface2

status_t MediaSync::setSurface(const sp<IGraphicBufferProducer> &output) {
...
    if (output != NULL) {
...
        // Try to connect to new output surface. If failed, current output surface will not
        // be changed.
        IGraphicBufferProducer::QueueBufferOutput queueBufferOutput;
        sp<OutputListener> listener(new OutputListener(this, output));
        IInterface::asBinder(output)->linkToDeath(listener);
        status_t status =
            output->connect(listener,
                            NATIVE_WINDOW_API_MEDIA,
                            true /* producerControlledByApp */,
                            &queueBufferOutput);
...
        if (mFrameScheduler == NULL) {
            mFrameScheduler = new VideoFrameScheduler();
            mFrameScheduler->init();
        }
    }
…
    //將MediaSync中的output和應用本身的surface關聯起來了
    mOutput = output;

    return NO_ERROR;
}

2.MediaSync自己創建出的新surface，其實直接看到最後那個
android_view_Surface_createFromIGraphicBufferProducer 就知道這裡新建立了一個surface了

static jobject android_media_MediaSync_createInputSurface(
        JNIEnv* env, jobject thiz) {
   ...
    sp<JMediaSync> sync = getMediaSync(env, thiz);
    ...
    // Tell the MediaSync that we want to use a Surface as input.
    sp<IGraphicBufferProducer> bufferProducer;
    //初始化bufferProducer, 詳見2.2
    status_t err = sync->createInputSurface(&bufferProducer);
    ...
    // Wrap the IGBP in a Java-language Surface.
    //利用bufferProducer, 建立新surface,詳見2.1
    return android_view_Surface_createFromIGraphicBufferProducer(env,
            bufferProducer);
}

2.1

jobject android_view_Surface_createFromIGraphicBufferProducer(JNIEnv* env,
        const sp<IGraphicBufferProducer>& bufferProducer) {
...
    sp<Surface> surface(new Surface(bufferProducer, true));
    if (surface == NULL) {
        return NULL;
    }
...
}

2.2

status_t MediaSync::createInputSurface(
        sp<IGraphicBufferProducer> *outBufferProducer) {
...
    sp<IGraphicBufferProducer> bufferProducer;
    sp<IGraphicBufferConsumer> bufferConsumer;
    BufferQueue::createBufferQueue(&bufferProducer, &bufferConsumer);

    sp<InputListener> listener(new InputListener(this));
    IInterface::asBinder(bufferConsumer)->linkToDeath(listener);
    status_t status =
        bufferConsumer->consumerConnect(listener, false /* controlledByApp */);
    if (status == NO_ERROR) {
        bufferConsumer->setConsumerName(String8("MediaSync"));
        // propagate usage bits from output surface
        mUsageFlagsFromOutput = 0;
        mOutput->query(NATIVE_WINDOW_CONSUMER_USAGE_BITS, &mUsageFlagsFromOutput);
        bufferConsumer->setConsumerUsageBits(mUsageFlagsFromOutput);
       //將MediaSync的input和新surface對應的producer關聯起來了
        *outBufferProducer = bufferProducer;
        mInput = bufferConsumer;
...
}

MediaSync avsync邏輯程式碼精讀

先來看video部分

1.

對video的調整主要在下面的方法中，可以看到，和NuPlayer中的相似度在95%以上
同樣可以看到對實際送顯時間的計算，以及利用VideoFrameScheduler進行調整

void MediaSync::onDrainVideo_l() {
...
    while (!mBufferItems.empty()) {
        int64_t nowUs = ALooper::GetNowUs();
        BufferItem *bufferItem = &*mBufferItems.begin();
        int64_t itemMediaUs = bufferItem->mTimestamp / 1000;
       //這裡就是呼叫MediaClock的getRealTimeFor方法，得到“視訊幀應該顯示的時間”
        int64_t itemRealUs = getRealTime(itemMediaUs, nowUs);

        // adjust video frame PTS based on vsync
        //利用vsync訊號調整itemRealUs
        itemRealUs = mFrameScheduler->schedule(itemRealUs * 1000) / 1000;
       //計算兩倍vsyncDuration
        int64_t twoVsyncsUs = 2 * (mFrameScheduler->getVsyncPeriod() / 1000);

        // post 2 display refreshes before rendering is due
        if (itemRealUs <= nowUs + twoVsyncsUs) {
            //如果當前時間距離itemRealUs已經不足2*vsyncDuration，則趕緊去顯示
            //將buffer的timeStamp設定為修正後的itemRealUs，前面我們分析NuPlayer的avsync時，曾經發現了一個問題，即NuPlayer實際送給render的video buffer timestamp並不是經過frameScheduler調整後的時間戳，看來在MediaSync中修正了這個問題
            bufferItem->mTimestamp = itemRealUs * 1000;
            bufferItem->mIsAutoTimestamp = false;

            if (mHasAudio) {
               //nowUs大於itemRealUs，說明視訊幀來晚了，這裡預設的門限值是40ms
                if (nowUs - itemRealUs <= kMaxAllowedVideoLateTimeUs) {
                    //queuebuffer給output，送去SF顯示
                    renderOneBufferItem_l(*bufferItem);
                } else {
                    // too late.丟幀
                    returnBufferToInput_l(
                            bufferItem->mGraphicBuffer, bufferItem->mFence);
                    mFrameScheduler->restart();
                }
            } else {
                // always render video buffer in video-only mode.
                renderOneBufferItem_l(*bufferItem);

                // smooth out videos >= 10fps
                mMediaClock->updateAnchor(
                        itemMediaUs, nowUs, itemMediaUs + 100000);
            }

            mBufferItems.erase(mBufferItems.begin());
            mNextBufferItemMediaUs = -1;
        } else {
            //如果當前時間距離itemRealUs大於2*vsyncDuration，則等到提前2*vsyncDuration的時候再去顯示
            if (mNextBufferItemMediaUs == -1
                    || mNextBufferItemMediaUs > itemMediaUs) {
                sp<AMessage> msg = new AMessage(kWhatDrainVideo, this);
                msg->post(itemRealUs - nowUs - twoVsyncsUs);
                mNextBufferItemMediaUs = itemMediaUs;
            }
            break;
        }
    }
}

2.

MediaSync的audio同步機制很有意思，是從java部分開始的, 儘管如此，我們還是可以發現他和NuPlayer的相似度在95%以上

public void queueAudio(
       @NonNull ByteBuffer audioData, int bufferId, long presentationTimeUs) {
...
   synchronized(mAudioLock) {
       mAudioBuffers.add(new AudioBuffer(audioData, bufferId, presentationTimeUs));
   }

   if (mPlaybackRate != 0.0) {
       postRenderAudio(0);
   }
}
//這裡同樣存在一個delay入參
private void postRenderAudio(long delayMillis) {
   mAudioHandler.postDelayed(new Runnable() {
       public void run() {
           synchronized(mAudioLock) {
               ...
               AudioBuffer audioBuffer = mAudioBuffers.get(0);
               int size = audioBuffer.mByteBuffer.remaining();
               // restart audio track after flush
               if (size > 0 && mAudioTrack.getPlayState() != AudioTrack.PLAYSTATE_PLAYING) {
                   try {
                       mAudioTrack.play();
                   } catch (IllegalStateException e) {
                       Log.w(TAG, "could not start audio track");
                   }
               }
               int sizeWritten = mAudioTrack.write(
                       audioBuffer.mByteBuffer,
                       size,
                       AudioTrack.WRITE_NON_BLOCKING);
               if (sizeWritten > 0) {
                   if (audioBuffer.mPresentationTimeUs != -1) {
                      //將pts時間送到framework中，用於更新currentPosition
                       native_updateQueuedAudioData(
                               size, audioBuffer.mPresentationTimeUs);
                       audioBuffer.mPresentationTimeUs = -1;
                   }

                   if (sizeWritten == size) {
                        //送給回撥函式，在回撥函式中進行audio的播放
                       postReturnByteBuffer(audioBuffer);
                       mAudioBuffers.remove(0);
                       if (!mAudioBuffers.isEmpty()) {
                           postRenderAudio(0);
                       }
                       return;
                   }
               }
               //和nuplayer中一樣，先計算出pendingDuration，然後等pendingDuration/2時間後再開始新的輪轉
               long pendingTimeMs = TimeUnit.MICROSECONDS.toMillis(
                       native_getPlayTimeForPendingAudioFrames());
               postRenderAudio(pendingTimeMs / 2);
           }
       }
   }, delayMillis);
}

2.1

到了framework中，對應的是下面的方法，可以說完全就是NuPlayer的avsync邏輯了，相信不用我說，你也能看懂

status_t MediaSync::updateQueuedAudioData(
        size_t sizeInBytes, int64_t presentationTimeUs) {
...

    int64_t numFrames = sizeInBytes / mAudioTrack->frameSize();
    int64_t maxMediaTimeUs = presentationTimeUs
            + getDurationIfPlayedAtNativeSampleRate_l(numFrames);

    int64_t nowUs = ALooper::GetNowUs();
    int64_t nowMediaUs = presentationTimeUs
            - getDurationIfPlayedAtNativeSampleRate_l(mNumFramesWritten)
            + getPlayedOutAudioDurationMedia_l(nowUs);

    mNumFramesWritten += numFrames;

    int64_t oldRealTime = -1;
    if (mNextBufferItemMediaUs != -1) {
        oldRealTime = getRealTime(mNextBufferItemMediaUs, nowUs);
    }

    mMediaClock->updateAnchor(nowMediaUs, nowUs, maxMediaTimeUs);
    mHasAudio = true;

    if (oldRealTime != -1) {
        int64_t newRealTime = getRealTime(mNextBufferItemMediaUs, nowUs);
        if (newRealTime >= oldRealTime) {
            return OK;
        }
    }

    mNextBufferItemMediaUs = -1;
    onDrainVideo_l();
    return OK;
}

至於其中關鍵的getPlayedOutAudioDurationMedia_l方法，也和NuPlayer中的完全一樣

int64_t MediaSync::getPlayedOutAudioDurationMedia_l(int64_t nowUs) {
...
    uint32_t numFramesPlayed;
    int64_t numFramesPlayedAt;
    AudioTimestamp ts;
    static const int64_t kStaleTimestamp100ms = 100000;

    status_t res = mAudioTrack->getTimestamp(ts);
    if (res == OK) {
        // case 1: mixing audio tracks.
        numFramesPlayed = ts.mPosition;
        numFramesPlayedAt =
            ts.mTime.tv_sec * 1000000LL + ts.mTime.tv_nsec / 1000;
        const int64_t timestampAge = nowUs - numFramesPlayedAt;
        if (timestampAge > kStaleTimestamp100ms) {
            // This is an audio FIXME.
            // getTimestamp returns a timestamp which may come from audio
            // mixing threads. After pausing, the MixerThread may go idle,
            // thus the mTime estimate may become stale. Assuming that the
            // MixerThread runs 20ms, with FastMixer at 5ms, the max latency
            // should be about 25ms with an average around 12ms (to be
            // verified). For safety we use 100ms.
            ALOGV("getTimestamp: returned stale timestamp nowUs(%lld) "
                  "numFramesPlayedAt(%lld)",
                  (long long)nowUs, (long long)numFramesPlayedAt);
            numFramesPlayedAt = nowUs - kStaleTimestamp100ms;
        }
        //ALOGD("getTimestamp: OK %d %lld",
        //      numFramesPlayed, (long long)numFramesPlayedAt);
    } else if (res == WOULD_BLOCK) {
        // case 2: transitory state on start of a new track
        numFramesPlayed = 0;
        numFramesPlayedAt = nowUs;
        //ALOGD("getTimestamp: WOULD_BLOCK %d %lld",
        //      numFramesPlayed, (long long)numFramesPlayedAt);
    } else {
        // case 3: transitory at new track or audio fast tracks.
        res = mAudioTrack->getPosition(&numFramesPlayed);
        CHECK_EQ(res, (status_t)OK);
        numFramesPlayedAt = nowUs;
        // MStar Android Patch Begin
        numFramesPlayedAt += 1000LL * mAudioTrack->latency() / 2; /* XXX */
        // MStar Android Patch End
        //ALOGD("getPosition: %d %lld", numFramesPlayed, (long long)numFramesPlayedAt);
    }

    //can't be negative until 12.4 hrs, test.
    //CHECK_EQ(numFramesPlayed & (1 << 31), 0);
    int64_t durationUs =
        getDurationIfPlayedAtNativeSampleRate_l(numFramesPlayed)
            + nowUs - numFramesPlayedAt;
    if (durationUs < 0) {
        // Occurs when numFramesPlayed position is very small and the following:
        // (1) In case 1, the time nowUs is computed before getTimestamp() is
        //     called and numFramesPlayedAt is greater than nowUs by time more
        //     than numFramesPlayed.
        // (2) In case 3, using getPosition and adding mAudioTrack->latency()
        //     to numFramesPlayedAt, by a time amount greater than
        //     numFramesPlayed.
        //
        // Both of these are transitory conditions.
        ALOGV("getPlayedOutAudioDurationMedia_l: negative duration %lld "
              "set to zero", (long long)durationUs);
        durationUs = 0;
    }
    ALOGV("getPlayedOutAudioDurationMedia_l(%lld) nowUs(%lld) frames(%u) "
          "framesAt(%lld)",
          (long long)durationUs, (long long)nowUs, numFramesPlayed,
          (long long)numFramesPlayedAt);
    return durationUs;
}

各位看官，如果您覺得本人的部落格對您有所幫助，可以掃描如下二維碼進行打賞，打賞多少您隨意~