http://www.mworkbox.com/wp/work/551.html

先看來自Live555官網的2個常見問題：
問題1：Why do most RTP sessions use separate streams for audio and video? How can a receiving client synchronize these streams?
回答：Sending audio and video in separate RTP streams provides a great deal of flexibility. For example, this makes it possible for a player to receive only the audio stream, but not video (or vice-versa). It would even be possible to have one computer receive and play audio, and a separate computer receive and play video.

These audio and video streams are synchronized using RTCP “Sender Report” (SR) packets – which map each stream’s RTP timestamp to ‘wall clock’ (NTP) time. For more information, see the IETF’s RTP/RTCP specification.

Receivers can then use this mapping to synchronize the incoming RTP streams. The LIVE555 Streaming Media code does this automatically: For subclasses of “RTPSource”, the “presentationTime” parameter that’s passed to the ‘afterGettingFunc’ of “getNextFrame()” (see “liveMedia/include/FramedSource.hh”) will be an accurate, time-synchronized time. (For this to work, you need to have also created a “RTCPInstance” for each RTP source.)

For example, if you use “openRTSP” to receive RTSP/RTP streams, then the contents of each RTP stream (audio and video) are written into separate files. This is done using the “FileSink” class. If you look at the “FileSink::afterGettingFrame()” member function, you’ll notice that there’s a “presentationTime” parameter for each incoming frame. Some other receiver could use the “presentationTime” parameter to synchronize audio and video.

問題2：But I notice that there’s an abrupt change in a stream’s presentation times after the first RTCP “SR” packet has been received. Is this a bug?
回答：No, this is normal, and expected; there’s no bug here. This happens because the first few presentation times – before RTCP synchronization occurs – are just ‘guesses’ made by the receiving code (based on the receiver’s ‘wall clock’ and the RTP timestamp). However, once RTCP synchronization occurs, all subsequent presentation times will be accurate.

This means is that a receiver should be prepared for the fact that the first few presentation times (until RTCP synchronization starts) will not be accurate. The code, however, can check this by calling “RTPSource:: hasBeenSynchronizedUsingRTCP()”. If this returns False, then the presentation times are not accurate, and should not be used for synchronization. However, once the call to returns True, then the presentation times (from then on) will be accurate.

我的心得：
1. Live555中關於寫檔案時，需要先同步音視訊流的例子見Live555的原始碼QuickTimeFileSink.cpp：

void QuickTimeFileSink
::afterGettingFrame(void* clientData, unsigned packetDataSize,
            unsigned numTruncatedBytes,
            struct timeval presentationTime,
            unsigned /*durationInMicroseconds*/) {
  SubsessionIOState* ioState = (SubsessionIOState*)clientData;
  if (!ioState->syncOK(presentationTime)) {
    // Ignore this data:音視訊還未同步，忽略這些資料包,syncOK中呼叫了hasBeenSynchronizedUsingRTCP()
    ioState->fOurSink.continuePlaying();
    return;
  }
  ...
  ioState->afterGettingFrame(packetDataSize, presentationTime);
}

2.1 通過Live555原始碼搜尋，查詢RTPSource同步標記(fCurPacketHasBeenSynchronizedUsingRTCP)設定的邏輯:
—- hasBeenSynchronizedUsingRTCP Matches (17 in 9 files) —-
MediaSession.cpp (livemedia): if (!rtpSource()->hasBeenSynchronizedUsingRTCP()) {
MultiFramedRTPSource.cpp (livemedia): fPresentationTime, fCurPacketHasBeenSynchronizedUsingRTCP,
RTPSource.cpp (livemedia): fCurPacketHasBeenSynchronizedUsingRTCP(False), fLastReceivedSSRC(0),
RTPSource.hh (livemedia\include): virtual Boolean hasBeenSynchronizedUsingRTCP();
RTPSource.hh (livemedia\include): Boolean fCurPacketHasBeenSynchronizedUsingRTCP;
testRTSPClient.cpp: if (fSubsession.rtpSource() != NULL && !fSubsession.rtpSource()->hasBeenSynchronizedUsingRTCP()) {
—- fHasBeenSyncedUsingRTCP Matches (3 in 2 files) —-
MultiFramedRTPSource.cpp (livemedia): fHasBeenSyncedUsingRTCP = hasBeenSyncedUsingRTCP;
MultiFramedRTPSource.cpp (livemedia): hasBeenSyncedUsingRTCP = fHasBeenSyncedUsingRTCP;
MultiFramedRTPSource.hh (livemedia\include): Boolean fHasBeenSyncedUsingRTCP;
—- assignMiscParams Matches (3 in 2 files) —-
MultiFramedRTPSource.cpp (livemedia): bPacket->assignMiscParams(rtpSeqNo, rtpTimestamp, presentationTime,
—- hasBeenSyncedUsingRTCP Matches (18 in 4 files) —-
MultiFramedRTPSource.cpp (livemedia): fHasBeenSyncedUsingRTCP = hasBeenSyncedUsingRTCP;
RTPSource.cpp (livemedia): resultHasBeenSyncedUsingRTCP = fHasBeenSynchronized;

2.2 真正進行RTCP時間同步標記判斷和儲存的地方，最終會通過bPacket->assignMiscParams函式，賦值給RTPSource::fCurPacketHasBeenSynchronizedUsingRTCP變數：

void MultiFramedRTPSource::networkReadHandler1() 
{
...
    struct timeval presentationTime; // computed by:
    Boolean hasBeenSyncedUsingRTCP; // computed by:
    receptionStatsDB()
      .noteIncomingPacket(rtpSSRC, rtpSeqNo, rtpTimestamp,
              timestampFrequency(),
              usableInJitterCalculation, presentationTime,
              hasBeenSyncedUsingRTCP, bPacket->dataSize());

    // Fill in the rest of the packet descriptor, and store it:
    struct timeval timeNow;
    gettimeofday(&timeNow, NULL);
    bPacket->assignMiscParams(rtpSeqNo, rtpTimestamp, presentationTime,
                  hasBeenSyncedUsingRTCP, rtpMarkerBit,
                  timeNow);
...
}

3.結論：參考Live55原始碼中的QuickTimeFileSink類中關於音視訊同步邏輯相關的部分，即可在自己的FileSink或者MemorySink中實現音視訊同步邏輯。

其他：研讀原始碼，發現MediaSubsession::getNormalPlayTime函式可以獲取當前rtp包的ntp時間，值得一試：
—- curPacketRTPTimestamp( Matches (3 in 2 files) —-
MediaSession.cpp (livemedia): u_int32_t timestampOffset = rtpSource()->curPacketRTPTimestamp() – rtpInfo.timestamp;
MediaSession.cpp (livemedia): u_int32_t timestampOffset = rtpSource()->curPacketRTPTimestamp() – rtpInfo.timestamp;
RTPSource.hh (livemedia\include): u_int32_t curPacketRTPTimestamp() const { return fCurPacketRTPTimestamp; }
—- getNormalPlayTime Matches (4 in 3 files) —-
MediaSession.cpp (livemedia):double MediaSubsession::getNormalPlayTime(struct timeval const& presentationTime) {
MediaSession.hh (livemedia\include): double getNormalPlayTime(struct timeval const& presentationTime);
MediaSession.hh (livemedia\include): double fNPT_PTS_Offset; // set by “getNormalPlayTime()”; add this to a PTS to get NPT
testRTSPClient.cpp: envir() << “\tNPT: ” << fSubsession.getNormalPlayTime(presentationTime);