摘要：國慶旅遊景點人太多，拍出來的照片全是人人人、車車車，該怎麼辦？不妨試試這個黑科技，讓你的出遊vlog秒變科幻大片。

本文分享自華為雲社群《國慶出遊神器，魔幻黑科技換天造物，讓vlog秒變科幻大片！》，作者：技術火炬手 。

國慶出遊，無論是拍人、拍景或是其他，“天空”都是關鍵元素。比如，一張平平無奇的景物圖加上落日餘暉的天空色調，氛圍感就有了。

當然，自然景觀的天空還不是最酷炫的。今天給大家介紹一款基於原生視訊的AI處理方法，不僅可以一鍵置換天空背景，還可以打造任意“天空之城”。

比如換成《星際迷航》中的浩瀚星空、宇宙飛船，將自己隨手拍的平平無奇vlog秒變為科幻大片，畫面毫無違和感。

該方法源自Github上的開源專案SkyAR，它可以自動識別天空，然後將天空從圖片中切割出來，再將天空替換成目標天空，從而實現魔法換天。

下面，我們將基於SkyAR和ModelArts的JupyterLab從零開始“換天造物”。只要腦洞夠大，利用這項AI技術，就可以創造出無限種玩法。

本案例在CPU和GPU下面均可執行，CPU環境執行預計花費9分鐘，GPU環境執行預計花費2分鐘。

實驗目標

通過本案例的學習：

瞭解影象分割的基本應用；

瞭解運動估計的基本應用；

瞭解影象混合的基本應用。

注意事項

  1. 如果您是第一次使用 JupyterLab，請檢視《ModelArts JupyterLab使用指導》瞭解使用方法；
  2. 如果您在使用 JupyterLab 過程中碰到報錯，請參考《ModelArts JupyterLab常見問題解決辦法》嘗試解決問題。

實驗步驟

1、安裝和匯入依賴包

  1. import os
  2. import moxing as mox
  3.  
  4. file_name = 'SkyAR'
  5. if not os.path.exists(file_name):
  6.     mox.file.copy('obs://modelarts-labs-bj4-v2/case_zoo/SkyAR/SkyAR.zip', 'SkyAR.zip')
  7.     os.system('unzip SkyAR.zip')
  8.     os.system('rm SkyAR.zip')
  9. mox.file.copy_parallel('obs://modelarts-labs-bj4-v2/case_zoo/SkyAR/resnet50-19c8e357.pth', '/home/ma-user/.cache/torch/checkpoints/resnet50-19c8e357.pth')
  10. INFO:root:Using MoXing-v1.17.3-43fbf97f
  11. INFO:root:Using OBS-Python-SDK-3.20.7
  12. !pip uninstall opencv-python -y
  13. !pip uninstall opencv-contrib-python -y
  14. Found existing installation: opencv-python 4.1.2.30
  15. Uninstalling opencv-python-4.1.2.30:
  16.   Successfully uninstalled opencv-python-4.1.2.30
  17. WARNING: Skipping opencv-contrib-python as it is not installed.
  18. !pip install opencv-contrib-python==4.5.3.56
  19. Looking in indexes: http://repo.myhuaweicloud.com/repository/pypi/simple
  20. Collecting opencv-contrib-python==4.5.3.56
  21.   Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/3f/ce/36772cc6d9061b423b080e86919fd62cdef0837263f29ba6ff92e07f72d7/opencv_contrib_python-4.5.3.56-cp37-cp37m-manylinux2014_x86_64.whl (56.1 MB)
  22.      |████████████████████████████████| 56.1 MB 166 kB/s eta 0:00:01|█████▋                          | 9.8 MB 9.4 MB/s eta 0:00:05 MB 9.4 MB/s eta 0:00:05███▏                | 26.6 MB 9.4 MB/s eta 0:00:04/s eta 0:00:03��██▍           | 35.8 MB 9.4 MB/s eta 0:00:03�███████████▌       | 42.9 MB 9.4 MB/s eta 0:00:02��██████████████▎   | 49.6 MB 166 kB/s eta 0:00:40
  23. Requirement already satisfied: numpy>=1.14.5 in /home/ma-user/anaconda3/envs/PyTorch-1.4/lib/python3.7/site-packages (from opencv-contrib-python==4.5.3.56) (1.20.3)
  24. Installing collected packages: opencv-contrib-python
  25. Successfully installed opencv-contrib-python-4.5.3.56
  26. WARNING: You are using pip version 20.3.3; however, version 21.1.3 is available.
  27. You should consider upgrading via the '/home/ma-user/anaconda3/envs/PyTorch-1.4/bin/python -m pip install --upgrade pip' command.
  28. cd SkyAR/
  29. /home/ma-user/work/Untitled Folder/SkyAR
  30. import time
  31. import json
  32. import base64
  33. import numpy as np
  34. import matplotlib.pyplot as plt
  35. import cv2
  36. import argparse
  37. from networks import *
  38. from skyboxengine import *
  39. import utils
  40. import torch
  41. from IPython.display import clear_output, Image, display, HTML
  42. %matplotlib inline
  43.  
  44. # 如果存在GPU則在GPU上面執行
  45. device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
  46. INFO:matplotlib.font_manager:generated new fontManager

2、預覽一下原視訊

  1. video_name = "test_videos/sky.mp4"
  2. def arrayShow(img):
  3.     img = cv2.resize(img, (0, 0), fx=0.25, fy=0.25, interpolation=cv2.INTER_NEAREST)
  4.     _,ret = cv2.imencode('.jpg', img)
  5.     return Image(data=ret)
  6.  
  7. # 開啟一個視訊流
  8. cap = cv2.VideoCapture(video_name)
  9.  
  10. frame_id = 0
  11. while True:
  12.     try:
  13.         clear_output(wait=True) # 清除之前的顯示
  14.         ret, frame = cap.read() # 讀取一幀圖片
  15.         if ret:
  16.             frame_id += 1
  17.             if frame_id > 200:
  18.                 break
  19.             cv2.putText(frame, str(frame_id), (5, 15), cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 255, 0), 1)  # 畫frame_id
  20.             tmp = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB) # 轉換色彩模式
  21.             img = arrayShow(frame)
  22.             display(img) # 顯示圖片
  23.             time.sleep(0.05) # 執行緒睡眠一段時間再處理下一幀圖片
  24.         else:
  25.             break
  26.     except KeyboardInterrupt:
  27.         cap.release()
  28. cap.release()

3、預覽一下要替換的天空圖片

  1. img= cv2.imread('skybox/sky.jpg')
  2. img2 = img[:,:,::-1]
  3. plt.imshow(img2)
  4. <matplotlib.image.AxesImage at 0x7fbea986c590>

4、自定義訓練引數

可以根據自己的需要, 修改下面的引數

skybox_center_crop: 天空體中心偏移

auto_light_matching: 自動亮度匹配

relighting_factor: 補光

recoloring_factor: 重新著色

halo_effect: 光環效應

  1. parameter = {
  2.   "net_G": "coord_resnet50",
  3.   "ckptdir": "./checkpoints_G_coord_resnet50",
  4.  
  5.   "input_mode": "video",
  6.   "datadir": "./test_videos/sky.mp4",
  7.   "skybox": "sky.jpg",
  8.  
  9.   "in_size_w": 384,
  10.   "in_size_h": 384,
  11.   "out_size_w": 845,
  12.   "out_size_h": 480,
  13.  
  14.   "skybox_center_crop": 0.5,
  15.   "auto_light_matching": False,
  16.   "relighting_factor": 0.8,
  17.   "recoloring_factor": 0.5,
  18.   "halo_effect": True,
  19.  
  20.   "output_dir": "./jpg_output",
  21.   "save_jpgs": False
  22. }
  23.  
  24. str_json = json.dumps(parameter)
  25. class Struct:
  26.     def __init__(self, **entries):
  27.         self.__dict__.update(entries)
  28. def parse_config():
  29.     data = json.loads(str_json)
  30.     args = Struct(**data)
  31.  
  32.     return args
  33. args = parse_config()
  34. class SkyFilter():
  35.  
  36.     def __init__(self, args):
  37.  
  38.         self.ckptdir = args.ckptdir
  39.         self.datadir = args.datadir
  40.         self.input_mode = args.input_mode
  41.  
  42.         self.in_size_w, self.in_size_h = args.in_size_w, args.in_size_h
  43.         self.out_size_w, self.out_size_h = args.out_size_w, args.out_size_h
  44.  
  45.         self.skyboxengine = SkyBox(args)
  46.  
  47.         self.net_G = define_G(input_nc=3, output_nc=1, ngf=64, netG=args.net_G).to(device)
  48.         self.load_model()
  49.  
  50.         self.video_writer = cv2.VideoWriter('out.avi',
  51.                                             cv2.VideoWriter_fourcc(*'MJPG'),
  52.                                             20.0,
  53.                                             (args.out_size_w, args.out_size_h))
  54.         self.video_writer_cat = cv2.VideoWriter('compare.avi',
  55.                                                 cv2.VideoWriter_fourcc(*'MJPG'),
  56.                                                 20.0,
  57.                                                 (2*args.out_size_w, args.out_size_h))
  58.  
  59.         if os.path.exists(args.output_dir) is False:
  60.             os.mkdir(args.output_dir)
  61.  
  62.         self.output_img_list = []
  63.  
  64.         self.save_jpgs = args.save_jpgs
  65.     def load_model(self):
  66.         # 載入預訓練的天空摳圖模型
  67.         print('loading the best checkpoint...')
  68.         checkpoint = torch.load(os.path.join(self.ckptdir, 'best_ckpt.pt'),
  69.                                 map_location=device)
  70.         self.net_G.load_state_dict(checkpoint['model_G_state_dict'])
  71.         self.net_G.to(device)
  72.         self.net_G.eval()
  73.     def write_video(self, img_HD, syneth):
  74.  
  75.         frame = np.array(255.0 * syneth[:, :, ::-1], dtype=np.uint8)
  76.         self.video_writer.write(frame)
  77.  
  78.         frame_cat = np.concatenate([img_HD, syneth], axis=1)
  79.         frame_cat = np.array(255.0 * frame_cat[:, :, ::-1], dtype=np.uint8)
  80.         self.video_writer_cat.write(frame_cat)
  81.  
  82.         # 定義結果緩衝區
  83.         self.output_img_list.append(frame_cat)
  84.     def synthesize(self, img_HD, img_HD_prev):
  85.  
  86.         h, w, c = img_HD.shape
  87.  
  88.         img = cv2.resize(img_HD, (self.in_size_w, self.in_size_h))
  89.  
  90.         img = np.array(img, dtype=np.float32)
  91.         img = torch.tensor(img).permute([2, 0, 1]).unsqueeze(0)
  92.  
  93.         with torch.no_grad():
  94.             G_pred = self.net_G(img.to(device))
  95.             G_pred = torch.nn.functional.interpolate(G_pred,
  96.                                                      (h, w),
  97.                                                      mode='bicubic',
  98.                                                      align_corners=False)
  99.             G_pred = G_pred[0, :].permute([1, 2, 0])
  100.             G_pred = torch.cat([G_pred, G_pred, G_pred], dim=-1)
  101.             G_pred = np.array(G_pred.detach().cpu())
  102.             G_pred = np.clip(G_pred, a_max=1.0, a_min=0.0)
  103.  
  104.         skymask = self.skyboxengine.skymask_refinement(G_pred, img_HD)
  105.  
  106.         syneth = self.skyboxengine.skyblend(img_HD, img_HD_prev, skymask)
  107.  
  108.         return syneth, G_pred, skymask
  109.     def cvtcolor_and_resize(self, img_HD):
  110.  
  111.         img_HD = cv2.cvtColor(img_HD, cv2.COLOR_BGR2RGB)
  112.         img_HD = np.array(img_HD / 255., dtype=np.float32)
  113.         img_HD = cv2.resize(img_HD, (self.out_size_w, self.out_size_h))
  114.  
  115.         return img_HD
  116.     def process_video(self):
  117.         # 逐幀處理視訊
  118.         cap = cv2.VideoCapture(self.datadir)
  119.         m_frames = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))
  120.         img_HD_prev = None
  121.  
  122.         for idx in range(m_frames):
  123.             ret, frame = cap.read()
  124.             if ret:
  125.                 img_HD = self.cvtcolor_and_resize(frame)
  126.  
  127.                 if img_HD_prev is None:
  128.                     img_HD_prev = img_HD
  129.  
  130.                 syneth, G_pred, skymask = self.synthesize(img_HD, img_HD_prev)
  131.  
  132.                 self.write_video(img_HD, syneth)
  133.  
  134.                 img_HD_prev = img_HD
  135.  
  136.                 if (idx + 1) % 50 == 0:
  137.                     print(f'processing video, frame {idx + 1} / {m_frames} ... ')
  138.  
  139.             else:  # 如果到達最後一幀
  140.                 break

5、替換天空

替換後輸出的視訊為out.avi，前後對比的視訊為compare.avi

  1. sf = SkyFilter(args)
  2. sf.process_video()
  3. initialize skybox...
  4. initialize network with normal
  5. loading the best checkpoint...
  6. processing video, frame 50 / 360 ...
  7. processing video, frame 100 / 360 ...
  8. no good point matched
  9. processing video, frame 150 / 360 ...
  10. processing video, frame 200 / 360 ...
  11. processing video, frame 250 / 360 ...
  12. processing video, frame 300 / 360 ...
  13. processing video, frame 350 / 360 ...

6、對比原視訊和替換後的視訊

  1. video_name = "compare.avi"
  2. def arrayShow(img):
  3.     _,ret = cv2.imencode('.jpg', img)
  4.     return Image(data=ret)
  5.  
  6. # 開啟一個視訊流
  7. cap = cv2.VideoCapture(video_name)
  8.  
  9. frame_id = 0
  10. while True:
  11.     try:
  12.         clear_output(wait=True) # 清除之前的顯示
  13.         ret, frame = cap.read() # 讀取一幀圖片
  14.         if ret:
  15.             frame_id += 1
  16.             cv2.putText(frame, str(frame_id), (5, 15), cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 255, 0), 1)  # 畫frame_id
  17.             tmp = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB) # 轉換色彩模式
  18.             img = arrayShow(frame)
  19.             display(img) # 顯示圖片
  20.             time.sleep(0.05) # 執行緒睡眠一段時間再處理下一幀圖片
  21.         else:
  22.             break
  23.     except KeyboardInterrupt:
  24.         cap.release()
  25. cap.release()

如果要生成自己的視訊，只要將test_videos中的sky.mp4視訊和skybox中的sky.jpg圖片替換成自己的視訊和圖片，然後重新一鍵執行就可以了。趕快來試一試吧，讓你的國慶大片更出彩！

華為雲社群祝大家國慶節快樂，度過一個開心的假期！

附錄

本案例源自華為雲AI Gallery：魔幻黑科技，可換天造物，秒變科幻大片！

點選關注，第一時間瞭解華為雲新鮮技術~