基於Kurento的WebRTC移動視訊群聊技術方案
阿新 • • 發佈：2019-01-08
說在前面的話：視訊實時群聊天有三種架構：
Mesh架構：終端之間互相連線，沒有中心伺服器，產生的問題，每個終端都要連線n-1個終端，每個終端的編碼和網路壓力都很大。群聊人數N不可能太大。
Router架構：終端之間引入中心伺服器，學名MCU(Multi Point Control Unit),每個終端的視訊流都發布到MCU伺服器上，然後伺服器負責編碼釋出多視訊流的工作，減輕客戶端的壓力。
Mix架構：在Router架構基礎上，多個視訊流在伺服器端被合為一個視訊流，減輕網路壓力。
下面講我們的選擇，在MCU方面有licode、kurento等解決方案。kurento在視訊群聊領域有專門的kurento Room解決方案，官方還提供一個kurento room server的樣例實現。
首先可以考慮不是一個Kurento Room Demo作為搭建方案原型的MCU元件。
Room Demo的部署可見:http://doc-kurento-room.readthedocs.io/en/stable/demo_deployment.html
其中碰到一些Maven編譯問題：
Unable to initialise extensions Component descriptor role: 'com.jcraft.jsch.UIKeyboardInteractive', implementation: 'org.apache.maven.wagon.providers.ssh.jsch.interactive.PrompterUIKeyboardInteractive', role hint: 'default' has a hint, but there are other implementations that don't
Maven的安裝版本需要時3.0以上
還有碰到找不到bower命令列問題。bower是node.js下面的一個包管理工具，安裝node.js以後用npm安裝即可
最後按照部署指南網頁中的命令啟動伺服器即可。
Demo伺服器有兩部分，一部分是Demo Web伺服器，二是把官方的kurento room server也整合到了這個demo中。不用再架設獨立的kurento room server
說說Android段的實施：再說一個公司：http://www.nubomedia.eu/，這家公司提供實時媒體通訊開源雲服務，核心元件可能是kurento media server，它的官網和kurento官網用一個模板，about裡面顯示兩家組織有聯絡，kurento官方提供的Java Client因為底層API原因在Android上不肯用，這個nubomedia組織提供了一個kurento android client的實現，同時還提供了一個kurento room client的實現以及room使用案例：https://github.com/nubomedia-vtt/nubo-test,這家公司對其開發的開源方案管理非常及時，早晨提個介面的issue，下午已經commit了程式碼修改。
這個案例雖然支援room溝通，但視訊溝通是基於room釋出訂閱機制做的雙人聊天。略改一下程式碼應該就可以實現多人聊天不過這家組織提供的兩個client實現和官方的介面高度相似。主要改的是PeerVideoActivity這個類，下面我share一個基本走通多端通訊的這個類的程式碼，供大家參考：
package fi.vtt.nubotest;
import android.app.ListActivity;
import android.content.SharedPreferences;
import android.graphics.PixelFormat;
import android.opengl.GLSurfaceView;
import android.os.Bundle;
import android.os.Handler;
import android.util.Log;
import android.view.Menu;
import android.view.MenuItem;
import android.view.View;
import android.view.WindowManager;
import android.widget.TextView;
import android.widget.Toast;
import org.webrtc.IceCandidate;
import org.webrtc.MediaStream;
import org.webrtc.PeerConnection;
import org.webrtc.RendererCommon;
import org.webrtc.SessionDescription;
import org.webrtc.VideoRenderer;
import org.webrtc.VideoRendererGui;
import java.util.Map;
import fi.vtt.nubomedia.kurentoroomclientandroid.RoomError;
import fi.vtt.nubomedia.kurentoroomclientandroid.RoomListener;
import fi.vtt.nubomedia.kurentoroomclientandroid.RoomNotification;
import fi.vtt.nubomedia.kurentoroomclientandroid.RoomResponse;
import fi.vtt.nubomedia.webrtcpeerandroid.NBMMediaConfiguration;
import fi.vtt.nubomedia.webrtcpeerandroid.NBMPeerConnection;
import fi.vtt.nubomedia.webrtcpeerandroid.NBMWebRTCPeer;
import fi.vtt.nubotest.util.Constants;
/**
* Activity for receiving the video stream of a peer
* (based on PeerVideoActivity of Pubnub's video chat tutorial example.
*/
public class PeerVideoActivity extends ListActivity implements NBMWebRTCPeer.Observer, RoomListener {
private static final String TAG = "PeerVideoActivity";
private NBMMediaConfiguration peerConnectionParameters;
private NBMWebRTCPeer nbmWebRTCPeer;
private SessionDescription localSdp;
private SessionDescription remoteSdp;
private String PaticipantID;
private VideoRenderer.Callbacks localRender;
private VideoRenderer.Callbacks remoteRender;
private GLSurfaceView videoView;
private SharedPreferences mSharedPreferences;
private int publishVideoRequestId;
private int sendIceCandidateRequestId;
private TextView mCallStatus;
private String username, calluser;
private boolean backPressed = false;
private Thread backPressedThread = null;
private static final int LOCAL_X_CONNECTED = 72;
private static final int LOCAL_Y_CONNECTED = 72;
private static final int LOCAL_WIDTH_CONNECTED = 25;
private static final int LOCAL_HEIGHT_CONNECTED = 25;
// Remote video screen position
private static final int REMOTE_X = 0;
private static int REMOTE_Y = 0;
private static final int REMOTE_WIDTH = 25;
private static final int REMOTE_HEIGHT = 25;
private Handler mHandler;
private CallState callState;
private enum CallState{
IDLE, PUBLISHING, PUBLISHED, WAITING_REMOTE_USER, RECEIVING_REMOTE_USER,PATICIPANT_JOINED,RECEIVING_PATICIPANT,
}
@Override
public void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
callState = CallState.IDLE;
setContentView(R.layout.activity_video_chat);
getWindow().addFlags(WindowManager.LayoutParams.FLAG_KEEP_SCREEN_ON);
mHandler = new Handler();
Bundle extras = getIntent().getExtras();
if (extras == null || !extras.containsKey(Constants.USER_NAME)) {
;
Toast.makeText(this, "Need to pass username to PeerVideoActivity in intent extras (Constants.USER_NAME).",
Toast.LENGTH_SHORT).show();
finish();
return;
}
this.username = extras.getString(Constants.USER_NAME, "");
Log.i(TAG, "username: " + username);
if (extras.containsKey(Constants.CALL_USER)) {
this.calluser = extras.getString(Constants.CALL_USER, "");
Log.i(TAG, "callUser: " + calluser);
}
this.mCallStatus = (TextView) findViewById(R.id.call_status);
TextView prompt = (TextView) findViewById(R.id.receive_prompt);
prompt.setText("Receive from " + calluser);
this.videoView = (GLSurfaceView) findViewById(R.id.gl_surface);
// Set up the List View for chatting
RendererCommon.ScalingType scalingType = RendererCommon.ScalingType.SCALE_ASPECT_FILL;
VideoRendererGui.setView(videoView, null);
localRender = VideoRendererGui.create( LOCAL_X_CONNECTED, LOCAL_Y_CONNECTED,
LOCAL_WIDTH_CONNECTED, LOCAL_HEIGHT_CONNECTED,
scalingType, true);
NBMMediaConfiguration.NBMVideoFormat receiverVideoFormat = new NBMMediaConfiguration.NBMVideoFormat(352, 288, PixelFormat.RGB_888, 20);
peerConnectionParameters = new NBMMediaConfiguration( NBMMediaConfiguration.NBMRendererType.OPENGLES,
NBMMediaConfiguration.NBMAudioCodec.OPUS, 0,
NBMMediaConfiguration.NBMVideoCodec.VP8, 0,
receiverVideoFormat,
NBMMediaConfiguration.NBMCameraPosition.FRONT);
nbmWebRTCPeer = new NBMWebRTCPeer(peerConnectionParameters, this, localRender, this);
nbmWebRTCPeer.initialize();
Log.i(TAG, "PeerVideoActivity initialized");
mHandler.postDelayed(publishDelayed, 4000);
MainActivity.getKurentoRoomAPIInstance().addObserver(this);
callState = CallState.PUBLISHING;
mCallStatus.setText("Publishing...");
}
private Runnable publishDelayed = new Runnable() {
@Override
public void run() {
nbmWebRTCPeer.generateOffer("derp", true);
}
};
@Override
public boolean onCreateOptionsMenu(Menu menu) {
// Inflate the menu; this adds items to the action bar if it is present.
getMenuInflater().inflate(R.menu.menu_video_chat, menu);
return true;
}
@Override
public boolean onOptionsItemSelected(MenuItem item) {
// Handle action bar item clicks here. The action bar will
// automatically handle clicks on the Home/Up button, so long
// as you specify a parent activity in AndroidManifest.xml.
int id = item.getItemId();
//noinspection SimplifiableIfStatement
if (id == R.id.action_settings) {
return true;
}
return super.onOptionsItemSelected(item);
}
@Override
protected void onStart() {
super.onStart();
}
@Override
protected void onPause() {
nbmWebRTCPeer.stopLocalMedia();
super.onPause();
}
@Override
protected void onResume() {
super.onResume();
nbmWebRTCPeer.startLocalMedia();
}
@Override
protected void onStop() {
endCall();
super.onStop();
}
@Override
protected void onDestroy() {
super.onDestroy();
}
@Override
public void onBackPressed() {
// If back button has not been pressed in a while then trigger thread and toast notification
if (!this.backPressed){
this.backPressed = true;
Toast.makeText(this,"Press back again to end.",Toast.LENGTH_SHORT).show();
this.backPressedThread = new Thread(new Runnable() {
@Override
public void run() {
try {
Thread.sleep(5000);
backPressed = false;
} catch (InterruptedException e){ Log.d("VCA-oBP","Successfully interrupted"); }
}
});
this.backPressedThread.start();
}
// If button pressed the second time then call super back pressed
// (eventually calls onDestroy)
else {
if (this.backPressedThread != null)
this.backPressedThread.interrupt();
super.onBackPressed();
}
}
public void hangup(View view) {
finish();
}
public void receiveFromRemote(View view){
Log.e(TAG,"--->receiveFromRemote");
if (callState == CallState.PUBLISHED){
callState = CallState.WAITING_REMOTE_USER;
nbmWebRTCPeer.generateOffer("remote", false);
runOnUiThread(new Runnable() {
@Override
public void run() {
mCallStatus.setText("Waiting remote stream...");
}
});
}
}
/**
* Terminates the current call and ends activity
*/
private void endCall() {
callState = CallState.IDLE;
try
{
if (nbmWebRTCPeer != null) {
nbmWebRTCPeer.close();
nbmWebRTCPeer = null;
}
}
catch (Exception e){e.printStackTrace();}
}
@Override
public void onLocalSdpOfferGenerated(final SessionDescription sessionDescription, NBMPeerConnection nbmPeerConnection) {
Log.e(TAG,"--->onLocalSdpOfferGenerated");
if (callState == CallState.PUBLISHING || callState == CallState.PUBLISHED) {
localSdp = sessionDescription;
Log.e(TAG,"--->onLocalSdpOfferGenerated:publish");
runOnUiThread(new Runnable() {
@Override
public void run() {
if (MainActivity.getKurentoRoomAPIInstance() != null) {
Log.d(TAG, "Sending " + sessionDescription.type);
publishVideoRequestId = ++Constants.id;
// String sender = calluser + "_webcam";
// MainActivity.getKurentoRoomAPIInstance().sendReceiveVideoFrom(sender, localSdp.description, publishVideoRequestId);
MainActivity.getKurentoRoomAPIInstance().sendPublishVideo(localSdp.description, false, publishVideoRequestId);
}
}
});
} else { // Asking for remote user video
Log.e(TAG,"--->onLocalSdpOfferGenerated:remote");
remoteSdp = sessionDescription;
// nbmWebRTCPeer.selectCameraPosition(NBMMediaConfiguration.NBMCameraPosition.BACK);
runOnUiThread(new Runnable() {
@Override
public void run() {
if (MainActivity.getKurentoRoomAPIInstance() != null) {
Log.e(TAG, "Sending--> " +calluser+ sessionDescription.type);
publishVideoRequestId = ++Constants.id;
String sender = calluser + "_webcam";
MainActivity.getKurentoRoomAPIInstance().sendReceiveVideoFrom(sender, remoteSdp.description, publishVideoRequestId);
}
}
});
}
}
@Override
public void onLocalSdpAnswerGenerated(SessionDescription sessionDescription, NBMPeerConnection nbmPeerConnection) {
}
@Override
public void onIceCandidate(IceCandidate iceCandidate, NBMPeerConnection nbmPeerConnection) {
Log.e(TAG,"--->onIceCandidate");
sendIceCandidateRequestId = ++Constants.id;
if (callState == CallState.PUBLISHING || callState == CallState.PUBLISHED){
Log.e(TAG,"--->onIceCandidate:publish");
MainActivity.getKurentoRoomAPIInstance().sendOnIceCandidate(this.username, iceCandidate.sdp,
iceCandidate.sdpMid, Integer.toString(iceCandidate.sdpMLineIndex), sendIceCandidateRequestId);
} else{
Log.e(TAG,"--->onIceCandidate:"+this.calluser);
MainActivity.getKurentoRoomAPIInstance().sendOnIceCandidate(this.calluser, iceCandidate.sdp,
iceCandidate.sdpMid, Integer.toString(iceCandidate.sdpMLineIndex), sendIceCandidateRequestId);
}
}
@Override
public void onIceStatusChanged(PeerConnection.IceConnectionState iceConnectionState, NBMPeerConnection nbmPeerConnection) {
Log.i(TAG, "onIceStatusChanged");
}
@Override
public void onRemoteStreamAdded(MediaStream mediaStream, NBMPeerConnection nbmPeerConnection) {
if (callState == CallState.PUBLISHING || callState == CallState.PUBLISHED) {
Log.e(TAG, "-->onRemoteStreamAdded-->no");
return;
}
Log.e(TAG, "-->onRemoteStreamAdded");
RendererCommon.ScalingType scalingType = RendererCommon.ScalingType.SCALE_ASPECT_FILL;
remoteRender = VideoRendererGui.create( REMOTE_X, REMOTE_Y,
REMOTE_WIDTH, REMOTE_HEIGHT,
scalingType, false);
REMOTE_Y = REMOTE_Y+25;
nbmWebRTCPeer.attachRendererToRemoteStream(remoteRender, mediaStream);
runOnUiThread(new Runnable() {
@Override
public void run() {
mCallStatus.setText("");
}
});
}
@Override
public void onRemoteStreamRemoved(MediaStream mediaStream, NBMPeerConnection nbmPeerConnection) {
Log.i(TAG, "onRemoteStreamRemoved");
}
@Override
public void onPeerConnectionError(String s) {
Log.e(TAG, "onPeerConnectionError:" + s);
}
@Override
public void onRoomResponse(RoomResponse response) {
Log.e(TAG, "-->OnRoomResponse:" + response);
if (Integer.valueOf(response.getId()) == publishVideoRequestId){
SessionDescription sd = new SessionDescription(SessionDescription.Type.ANSWER,
response.getValue("sdpAnswer").get(0));
if (callState == CallState.PUBLISHING){
callState = CallState.PUBLISHED;
nbmWebRTCPeer.processAnswer(sd, "derp");
} else if (callState == CallState.WAITING_REMOTE_USER){
callState = CallState.RECEIVING_REMOTE_USER;
nbmWebRTCPeer.processAnswer(sd, "remote");
} else if (callState == CallState.PATICIPANT_JOINED){
callState = CallState.RECEIVING_PATICIPANT;
nbmWebRTCPeer.processAnswer(sd, this.PaticipantID);
//NOP
}
}
}
@Override
public void onRoomError(RoomError error) {
Log.e(TAG, "OnRoomError:" + error);
}
@Override
public void onRoomNotification(RoomNotification notification) {
Log.e(TAG, "OnRoomNotification--> (state=" + callState.toString() + "):" + notification);
if(notification.getMethod().equals("iceCandidate")) {
Map<String, Object> map = notification.getParams();
String sdpMid = map.get("sdpMid").toString();
int sdpMLineIndex = Integer.valueOf(map.get("sdpMLineIndex").toString());
String sdp = map.get("candidate").toString();
IceCandidate ic = new IceCandidate(sdpMid, sdpMLineIndex, sdp);
Log.e(TAG, "callState-->" + callState);
if (callState == CallState.PUBLISHING || callState == CallState.PUBLISHED) {
nbmWebRTCPeer.addRemoteIceCandidate(ic, "derp");
}else if(callState==CallState.PATICIPANT_JOINED || callState== CallState.RECEIVING_PATICIPANT){
nbmWebRTCPeer.addRemoteIceCandidate(ic,this.PaticipantID);
}else {
nbmWebRTCPeer.addRemoteIceCandidate(ic, "remote");
}
}
if(notification.getMethod().equals("participantPublished"))
{
Map<String, Object> map = notification.getParams();
final String user = map.get("id").toString();
this.calluser = user;
this.PaticipantID = "pt_"+this.calluser;
PeerVideoActivity.this.runOnUiThread(new Runnable() {
@Override
public void run() {
callState = CallState.PATICIPANT_JOINED;
nbmWebRTCPeer.generateOffer(PaticipantID, false);
}
});
}
}
@Override
public void onRoomConnected() {
}
@Override
public void onRoomDisconnected() {
}
}再就是android room demo中的MainActivity的新增cert的程式碼要去掉註釋，讓這段程式碼生效，就可以連通伺服器了。
在iOS的實施方面，上面這家公司也提供了一個工具包：https://github.com/nubomediaTI/Kurento-iOS ，工具包裡面也有demo
Web方面，最上面官方的哪個demo就足夠參考了
後記：很榮幸這篇部落格獲得了很多CSDN程式設計師的關注和詢問，這隻能證明我很榮幸有機會在去年的那個時間點(16年7月)在大家之前處理了一個後續大家都很關注的技術問題，而處理這個問題主要用到的伺服器端room server專案和android端nubo test專案，官方在後續好像都做了一定的升級，反而是我自己搞完這個之後，因為產品設計的原因，後來再沒有深入地去生產實施這個東西，甚至開發筆記本關於這個專案的原始碼專案好像都已經刪除了，對於大家提出的問題，早期的我還能答一答，後面的我估計你們用到的原始碼和我用到的原始碼估計都不是一個版本了，再就是裡面的程式碼細節也基本忘得差不多，在這兒我建議後續開發這個功能可以去深入閱讀分析Kurento官方(https://github.com/Kurento）和歐洲媒體服務雲服務商nubomedia官方（https://github.com/nubomedia-vtt）的程式碼示例和文件。我面給出的程式碼樣例是基於nubomedia一對一視聊樣例改的，官方原始程式碼樣例在這段時間內都有了變更。在掌握大的基本WebRTC通訊的原理的前提下，我覺得改新的程式碼估計也不會太難。