一、背景

      最近有一個需求，使用android系統的裝置，從IP攝像頭（RTSP SERVER）獲取到的視訊中的每一幀進行處理（人臉檢測），直接使用ffmpeg進行實現比較簡單，但是苦於對ffmpeg不太熟悉，獲取到的視訊延遲較高，只好轉戰看看LibVLC能否獲得更好的效果。

        兩篇文章幫助較大，在此感謝：

二、主要思路

     主要思路是利用libVLC 現有的API：libvlc_video_set_format()  、libvlc_video_set_callbacks() 對libvlc解碼後的幀進行擷取，然後把設定callback的介面通過JNI暴露給JAVA層。這兩個函式都在libvlc_media_player.h中有具體的說明。下面分別進行分析。

三、 具體實現

3.1 Lib層

3.1.1 libvlc_video_set_format() ：

/**
 * Set decoded video chroma and dimensions.
 * This only works in combination with libvlc_video_set_callbacks(),
 * and is mutually exclusive with libvlc_video_set_format_callbacks().
 *
 * \param mp the media player
 * \param chroma a four-characters string identifying the chroma
 *               (e.g. "RV32" or "YUYV")
 * \param width pixel width
 * \param height pixel height
 * \param pitch line pitch (in bytes)
 * \version LibVLC 1.1.1 or later
 * \bug All pixel planes are expected to have the same pitch.
 * To use the YCbCr color space with chrominance subsampling,
 * consider using libvlc_video_set_format_callbacks() instead.
 */
LIBVLC_API
void libvlc_video_set_format( libvlc_media_player_t *mp, const char *chroma,
                              unsigned width, unsigned height,
                              unsigned pitch );
 

該函式的作用是設定你希望的輸出幀的格式和尺寸，libvlc會非常賣力地幫你解碼，然後轉換成你希望的格式。

· chroma為視訊輸出幀格式的選擇，可以是常用的YUYV、NV12、RGBA等各種格式；

· width、height為你希望輸出的幀的長寬；

· pitch 直接的翻譯是“場”，它的值是 幀寬度 × 每個畫素所佔的位元組數，當然每個畫素的位元組數是由你選擇的幀格式決定的，比如RGBA一個畫素4個位元組，YUYV一個畫素1.5個位元組。具體自己問google。

3.1.2 libvlc_video_set_callbacks()：

/**
 * Set callbacks and private data to render decoded video to a custom area
 * in memory.
 * Use libvlc_video_set_format() or libvlc_video_set_format_callbacks()
 * to configure the decoded format.
 *
 * \warning Rendering video into custom memory buffers is considerably less
 * efficient than rendering in a custom window as normal.
 *
 * For optimal perfomances, VLC media player renders into a custom window, and
 * does not use this function and associated callbacks. It is <b>highly
 * recommended</b> that other LibVLC-based application do likewise.
 * To embed video in a window, use libvlc_media_player_set_xid() or equivalent
 * depending on the operating system.
 *
 * If window embedding does not fit the application use case, then a custom
 * LibVLC video output display plugin is required to maintain optimal video
 * rendering performances.
 *
 * The following limitations affect performance:
 * - Hardware video decoding acceleration will either be disabled completely,
 *   or require (relatively slow) copy from video/DSP memory to main memory.
 * - Sub-pictures (subtitles, on-screen display, etc.) must be blent into the
 *   main picture by the CPU instead of the GPU.
 * - Depending on the video format, pixel format conversion, picture scaling,
 *   cropping and/or picture re-orientation, must be performed by the CPU
 *   instead of the GPU.
 * - Memory copying is required between LibVLC reference picture buffers and
 *   application buffers (between lock and unlock callbacks).
 *
 * \param mp the media player
 * \param lock callback to lock video memory (must not be NULL)
 * \param unlock callback to unlock video memory (or NULL if not needed)
 * \param display callback to display video (or NULL if not needed)
 * \param opaque private pointer for the three callbacks (as first parameter)
 * \version LibVLC 1.1.1 or later
 */
LIBVLC_API
void libvlc_video_set_callbacks( libvlc_media_player_t *mp,
                                 libvlc_video_lock_cb lock,
                                 libvlc_video_unlock_cb unlock,
                                 libvlc_video_display_cb display,
                                 void *opaque );
 

這個函式是用來設定在libvlc解碼過程中的三個回撥的，觸發的時機分別為：

·libvlc_video_lock_cb - 有一個新的幀要開始解碼；

·libvlc_video_unlock_cb - 當前幀解碼完成；

·libvlc_video_display_cb - 開始顯示。

PS：需要注意的是，在我們自己呼叫這些這個函式設定這些回撥時，其他流程中設定的回撥就會失效。比如在java中呼叫ivlcVout.setVideoView(textureView)設定輸出的View時，實際上是會對這三個回撥進行設定的，而在這之後手動呼叫 libvlc_video_set_callbacks（），就會對這些回撥進行覆蓋，使之不在會在MediaPlayer.play時，顯示在View中。如果還需要顯示在View中，則需要進行其他的處理。

3.2 JNI層

void
Java_org_videolan_libvlc_MediaPlayer_nativeSetVideoFormat(JNIEnv *env, jobject thiz, 
   jstring format, jint width, jint height, jint pitch) {

   vlcjni_object *p_obj = VLCJniObject_getInstance(env, thiz);

   if (!p_obj)
       return;

   const char *formatStr = (*env)->GetStringUTFChars(env, format, NULL);

   libvlc_video_set_format(p_obj->u.p_mp, formatStr, width, height, pitch);

   (*env)->ReleaseStringUTFChars(env, format, formatStr);
}

struct myfield {
   jclass mediaPlayerClazz;
   jmethodID onDisplayCallback;
   jobject thiz;
   void *buffer;
} myfield;

static void *lock(void *data, void ** p_pixels) {
   *p_pixels = myfield.buffer;
   return NULL;
}

static void unlock(void *data, void *id, void * const * p_pixels) {
}

static pthread_mutex_t myMutex = PTHREAD_MUTEX_INITIALIZER;

static void display(void *data, void *id) {
   JavaVM *jvm = fields.jvm; //這個地方需要修改 "utils.h" 中的struct fields，增加成員JavaVM* jvm,並且在libvlcjni.c的中進行賦值
   JNIEnv *env; 
   int stat = (*jvm)->GetEnv(jvm, (void **)&env, JNI_VERSION_1_2);
   if (stat == JNI_EDETACHED) {
       if ((*jvm)->AttachCurrentThread(jvm, (void **) &env, NULL) != 0) {
           return;
       }
   } else if (stat == JNI_OK) {
       //
   } else if (stat == JNI_EVERSION) {
       return;
   }

   pthread_mutex_lock(&myMutex);

   if (myfield.thiz != NULL) {
       (*env)->CallVoidMethod(env, myfield.thiz, myfield.onDisplayCallback);
   }

   pthread_mutex_unlock(&myMutex);

   (*jvm)->DetachCurrentThread(jvm);
}

void
Java_org_videolan_libvlc_MediaPlayer_nativeSetVideoBuffer(JNIEnv *env, jobject thiz, jobject buffer) {
   vlcjni_object *p_obj = VLCJniObject_getInstance(env, thiz);

   libvlc_media_player_t *mp = p_obj->u.p_mp;
   if (!mp) {
       return;
   }

   if (buffer == NULL) {
       (*env)->DeleteGlobalRef(env, myfield.mediaPlayerClazz);
       pthread_mutex_lock(&myMutex);
       (*env)->DeleteGlobalRef(env, myfield.thiz);
       myfield.thiz = NULL;
       pthread_mutex_unlock(&myMutex);
       return;
   }

   myfield.mediaPlayerClazz = (*env)->FindClass(env, "org/videolan/libvlc/MediaPlayer");
   myfield.mediaPlayerClazz = (jclass) (*env)->NewGlobalRef(env, myfield.mediaPlayerClazz);
   myfield.onDisplayCallback = (*env)->GetMethodID(env, myfield.mediaPlayerClazz, "onDisplay", "()V");
   myfield.thiz = (*env)->NewGlobalRef(env, thiz);
   myfield.buffer = (*env)->GetDirectBufferAddress(env, buffer);

   libvlc_video_set_callbacks(mp, lock, NULL, display, NULL);
}

在libvlcjni.c的中對fields.jvm進行賦值

int VLCJNI_OnLoad(JavaVM *vm, JNIEnv* env)
{
    myVm = vm;
    fields.jvm = vm;
    /* Create a TSD area and setup a destroy callback when a thread that
     * previously set the jni_env_key is canceled or exited */
    if (pthread_key_create(&jni_env_key, jni_detach_thread) != 0)
        return -1;
    ...
}

3.3 JAVA層

2.3.1 修改org/videolan/libvlc/MediaPlayer.java

增加JNI介面：

    private native void nativeSetVideoFormat(String format, int width, int height, int pitch);
    private native void nativeSetVideoBuffer(ByteBuffer buffer);

增加設定幀格式及回撥函式：

    public void setVideoFormat(String format, int width, int height, int pitch){
        nativeSetVideoFormat(format, width, height, pitch);
    }

    public void setVideoCallback(ByteBuffer buffer, MediaPlayerCallback callback) {
        mBuffer = buffer;
        mCallback = callback;
        nativeSetVideoBuffer(buffer);
    }

2.3.2 Activity中呼叫

        LibVLC mLibVLC = new LibVLC(this);

        Uri uri = Uri.parse("http://www.w3school.com.cn/i/movie.ogg");
        Media m = new Media(mLibVLC, uri);

        final int FRM_WIDTH = 500;
        final int FRM_HEIGHT = 400;
        final int PIXEL_SIZE = 4;

        MediaPlayer mediaPlayer = new MediaPlayer(m);
        ByteBuffer frameBuffer = ByteBuffer.allocateDirect(FRM_WIDTH*FRM_HEIGHT*PIXEL_SIZE);
        mediaPlayer.setVideoFormat("RGBA",FRM_WIDTH,FRM_HEIGHT,FRM_WIDTH*PIXEL_SIZE);
        mediaPlayer.setVideoCallback(frameBuffer, new MediaPlayerCallback() {
            @Override
            public void onDisplay(ByteBuffer buffer) {
                buffer.rewind();
                final Bitmap bitmap = Bitmap.createBitmap(FRM_WIDTH,FRM_HEIGHT, Bitmap.Config.ARGB_8888);
                bitmap.copyPixelsFromBuffer(buffer);
                runOnUiThread(new Runnable() {
                    @Override
                    public void run() {
                        imageView.setImageBitmap(bitmap);
                    }
                });
            }
        });

        mediaPlayer.setMedia(m);
        mediaPlayer.play();

至此就大功告成，可以獲取到視訊中的每一幀了。

四、存在的問題

4.1 解碼效率問題

有心的看官可能已經注意到了這段話，來看下官方對於libvlc_video_set_callbacks介紹，出自全知全能的

 * The following limitations affect performance:
 * - Hardware video decoding acceleration will either be disabled completely,
 *   or require (relatively slow) copy from video/DSP memory to main memory.
 * - Sub-pictures (subtitles, on-screen display, etc.) must be blent into the
 *   main picture by the CPU instead of the GPU.
 * - Depending on the video format, pixel format conversion, picture scaling,
 *   cropping and/or picture re-orientation, must be performed by the CPU
 *   instead of the GPU.
 * - Memory copying is required between LibVLC reference picture buffers and
 *   application buffers (between lock and unlock callbacks).

呼叫了libvlc_video_set_callbacks相當於在人家原有的完美的軟硬體結合的流程中橫叉了一槓子，變純軟體解碼了，字幕也需要CPU混合進影象了，video format的各種轉換也用CPU了，GPU罷工了，等等效率問題。其中的酸甜苦辣就只能根據應用場景自行判斷了。

4.2 顯示效率問題

像我的程式碼中這樣，把每一幀轉換成Bitmap進行顯示，就有一次額外的記憶體拷貝的過程，肯定不是最優方案，這個還沒細研究，估計是有解，小弟孤陋寡聞，現在還不知道怎麼搞。