Octavia 的 HTTPS 與自建、簽發 CA 證書
阿新 • • 發佈：2018-12-16
目錄
文章目錄
Octavia 為什麼需要自建 CA 證書？
Note: For production use the ca issuing the client certificate and the ca issuing the server certificate need to be different so a hacker can’t just use the server certificate from a compromised amphora to control all the others.
可以想象，如果 octavia-worker、amphora 通訊使用的證書與 User、Dashboard 通訊使用的證書相同，那麼 User 進出 amphora 就如入無人之地。簡而言之，Octavia 自建 CA 證書是為了防止惡意使用者登入並利用 amphora 作為 “肉雞” 來攻擊 OpenStack 的內部網路。
Octavia 提供了指令碼 octavia/bin/create_certificates.sh 和配置檔案 octavia/etc/certificates/openssl.cnf，只需執行下述指令即可完成自建 CA。
$ source /opt/rocky/octavia/bin/create_certificates.sh /etc/octavia/certs/ /opt/rocky/octavia/etc/certificates/openssl.cnf $ ll /etc/octavia/certs/ total 44 -rw-r--r-- 1 stack stack 1294 Oct 26 12:51 ca_01.pem -rw-r--r-- 1 stack stack 989 Oct 26 12:51 client.csr -rw-r--r-- 1 stack stack 1708 Oct 26 12:51 client.key -rw-r--r-- 1 stack stack 4405 Oct 26 12:51 client-.pem -rw-r--r-- 1 stack stack 6113 Oct 26 12:51 client.pem -rw-r--r-- 1 stack stack 71 Oct 26 12:51 index.txt -rw-r--r-- 1 stack stack 21 Oct 26 12:51 index.txt.attr -rw-r--r-- 1 stack stack 0 Oct 26 12:51 index.txt.old drwxr-xr-x 2 stack stack 20 Oct 26 12:51 newcerts drwx------ 2 stack stack 23 Oct 26 12:51 private -rw-r--r-- 1 stack stack 3 Oct 26 12:51 serial -rw-r--r-- 1 stack stack 3 Oct 26 12:51 serial.old
- ca_01.pem：CA 證書檔案
- client.csr：Server CSR 證書籤名請求檔案
- client.key：Server 私鑰檔案
- client-.pem：PEM 編碼的 Server 證書檔案
- client.pem：結合了 client-.pem 和 client.key 的檔案
與 CA 認證相關的主要有以下配置：
[certificates] ca_private_key_passphrase = foobar ca_private_key = /etc/octavia/certs/private/cakey.pem ca_certificate = /etc/octavia/certs/ca_01.pem [haproxy_amphora] server_ca = /etc/octavia/certs/ca_01.pem client_cert = /etc/octavia/certs/client.pem [controller_worker] client_ca = /etc/octavia/certs/ca_01.pem
-
[haproxy_amphora]section 在 octavia-worker AmphoraAPIClient 被應用，AmphoraAPIClient 拿著 client.pem（包含 Server 證書、Server 私鑰）和 CA 公鑰向 amphora-agent 發起 SSL 通訊。 -
[certificates]section 應用在 create amphora flow 的 GenerateServerPEMTask 中，指定 CA 中心為 amphora 簽署服務端證書。 -
[controller_worker]的 client_ca 在 Task:CertComputeCreate 中被應用，指定了 CA 證書路徑。
下面主要圍繞這 3 個 section 來進行分析。
GenerateServerPEMTask
從 UML 圖可以看出當 octavia-worker 需要使用 REST 方式與 amphora-agent 通訊時 create_amphora_flow 才會載入 GenerateServerPEMTask 用於簽發一張 PEM 編碼的 Amphora 證書用於建立 HTTPS 通訊。
class GenerateServerPEMTask(BaseCertTask):
"""Create the server certs for the agent comm
Use the amphora_id for the CN
"""
def execute(self, amphora_id):
cert = self.cert_generator.generate_cert_key_pair(
cn=amphora_id,
validity=CERT_VALIDITY)
return cert.certificate + cert.private_key
octavia.certificates 實現了 local_cert_generator(default) 和 anchor_cert_generator 兩種證書生成器，通過配置 [certificates] cert_generator 選定，這裡以 local_cert_generator 為例：
# file: /opt/rocky/octavia/octavia/certificates/generator/local.py
@classmethod
def generate_cert_key_pair(cls, cn, validity, bit_length=2048,
passphrase=None, **kwargs):
pk = cls._generate_private_key(bit_length, passphrase)
csr = cls._generate_csr(cn, pk, passphrase)
cert = cls.sign_cert(csr, validity, **kwargs)
cert_object = local_common.LocalCert(
certificate=cert,
private_key=pk,
private_key_passphrase=passphrase
)
return cert_object
method generate_cert_key_pair 的語義為：
- 生成 Amphora 私鑰
- 生成 Amphora 證書籤名請求（CSR）
- 向 CA 中心申請簽署 Amphora 證書
區別於 create_certificates.sh 指令碼的 openssl 指令，octavia-worker 使用了內建的 cryptography 庫來實現。GenerateServerPEMTask 的是 Amphora 的 server.pem，包含了 Amphora 證書（公鑰）和 Amphora 私鑰。
(Pdb) cert
<octavia.certificates.common.local.LocalCert object at 0x7ff1b879ee50>
(Pdb) cert.certificate + cert.private_key
'-----BEGIN CERTIFICATE-----\nMIIDiTCCAnGgAwIBAgIRAO8gS0SKikCiuMCRUiKkLYUwDQYJKoZIhvcNAQELBQAw\nXDELMAkGA1UEBhMCVVMxDzANBgNVBAgMBkRlbmlhbDEUMBIGA1UEBwwLU3ByaW5n\nZmllbGQxDDAKBgNVBAoMA0RpczEYMBYGA1UEAwwPd3d3LmV4YW1wbGUuY29tMB4X\nDTE4MTExNDA2MjY0NVoXDTIwMTExMzA2MjY0NVowLzEtMCsGA1UEAwwkZjQ5M2M5\nMDQtZWFhMy00NTljLThjNzctMTU2OGYyOWMwYzI3MIIBIjANBgkqhkiG9w0BAQEF\nAAOCAQ8AMIIBCgKCAQEA2spT4jGxt1nrfRwFburC2ErkdPaO3KE15ndoc17Blw+h\ngtLKhMhqCGGmWx7h9XsLV98JtglYuJDwh2vI7iToqN22izIhyUfef9j6asgUDs4K\nAB8502uPPeVwxmB60KeZQzES2PxgxpngUeMktpMv90/YIyJyEPgeEKj4/41C8Pr0\nMwvQVfJD28O2A/S5GgJWWjSgZA3vvEVflyhe0nnCWdJNwSFSzoR68IAF8RCBOm2e\n0UAYS/p9nabvkCPxpO8fhNbqzg/NCwIb+Vi06mC3L0tzyApI+Oks1Jd7uMM4O/BV\nvzD68Me/KbirYr6JEUHYAlIvyQ51ljAC0nWWJviW1wIDAQABo3MwcTAMBgNVHRMB\nAf8EAjAAMC8GA1UdEQQoMCaCJGY0OTNjOTA0LWVhYTMtNDU5Yy04Yzc3LTE1Njhm\nMjljMGMyNzAOBgNVHQ8BAf8EBAMCA7gwIAYDVR0lAQH/BBYwFAYIKwYBBQUHAwEG\nCCsGAQUFBwMCMA0GCSqGSIb3DQEBCwUAA4IBAQAgP+yoiW/Otfl9Sx/VBh2Ehw0H\n+82rPT7Yvs8UfnOv/AqDiUzOZMbmJQNfwMgzwY9l7yY4h0DMQsTortYHCuOMTaWB\nIsPRCCZeFtSxgYnHjW8Ggw1bEHNUWlkazENzA/2fKNnAkj/ddcr26TOADTGi4wB1\nVqvU51+UckA+Qn19fDdmlYWhRDhh3ye1wgQNZmIGiBoFuwJpQBBYST9AS+xcr80g\ntCgThHgjqTMr0I3K1Dx/zMA+/eTIWb4e8T7jcd0iW7fTFtux5/NVjcTWjcAFvTAd\nZ9mNB5DTKrc3V61AybpIyZeI545Had0GM+SdHXNzFI2tCtgTvT/ZEAsqRiRv\n-----END CERTIFICATE-----\n-----BEGIN RSA PRIVATE KEY-----\nMIIEpQIBAAKCAQEA2spT4jGxt1nrfRwFburC2ErkdPaO3KE15ndoc17Blw+hgtLK\nhMhqCGGmWx7h9XsLV98JtglYuJDwh2vI7iToqN22izIhyUfef9j6asgUDs4KAB85\n02uPPeVwxmB60KeZQzES2PxgxpngUeMktpMv90/YIyJyEPgeEKj4/41C8Pr0MwvQ\nVfJD28O2A/S5GgJWWjSgZA3vvEVflyhe0nnCWdJNwSFSzoR68IAF8RCBOm2e0UAY\nS/p9nabvkCPxpO8fhNbqzg/NCwIb+Vi06mC3L0tzyApI+Oks1Jd7uMM4O/BVvzD6\n8Me/KbirYr6JEUHYAlIvyQ51ljAC0nWWJviW1wIDAQABAoIBAEAJpE+6V9fgm8p8\nnyJ92BXSpdeOKvZswQf5vzq1a1g5nP5bkCcZOd/GJRjaiyx8nS9U+tSrG6q50Yzx\ngVgiuW5jpoBLZhQx0u/8pB8I/MXwjIDIovY8rypgs4d8ybW0uGkwPeIAzJqUg1G0\neBRwNEPgvNRbyqMo3DPoISk7QXKilmk/h5em1KmodBU9YtgIqi8F+Rs60csOPo5Z\nX70YGq5dICN1xIqW8eqzvbVO4JF/oU7JqhEyfcG6S029Ilj29bAPCRLEMZIUxeP5\ncMjGVP59PCt1z42UEnaxLEmzIpD2vVq06xEC3/0BvLLGTY6FOfzXjeEt3i/fdcUq\nLiNIejECgYEA/QxXwmPu9c9v4uUqhMcG6AUJd1Hlrfq/gJIaNRGsp/UtK4qtPMgw\n6GTv9Sv1D4Vuad65recdrGXYgAPWj1TRbIqimBE0Yu0z5vuAfScu7yrrZsWZXHJO\nVIP2310ZPnJFXwRSCA1VpIM8MoKfoAfpAMfDr7PM0gEjDapPpFLsl4UCgYEA3Veu\n8fTqqgxCWFe0IRZRiRT5FciRupBvuOVmYMIqIGtSD/l21dOQ0rd5o5ZBZsInO80y\ndXWk6YEni6dqSjxYwlaTHfG3ZyZlSEeaSHEhGiLvItxaMh6DJGcbTKUgkDiHFf5K\nW/KpJ91C4ByykHZmGYesz/z4Z8vadjPrf8gpLasCgYEAoJHedjlXfp88jiuAyXRJ\ni5z2nsJXDgkYz4rmGlq2xnUrTn/W4cTeU/kI0vgrrseqgn+ULyeCisytjr3gvl7B\n7TAjcH8qUMPXtXBN3hypCZagfTxRznmx/qsmUiIPTLLSFjL1oqpjd9rWre55P+EF\nFzurjqh3BaM3DQrPMqR0AMkCgYEAkt5BqS7YHulvhGr9jQ7gH1OZS8kAWYjJeShO\nXFm51jUgCJWBMrTlXcx8m/1xfBvMKLQpjSL4wDAA63u03XlZc+o6SB5BkeI6RlGs\nn/DhBBS2FK2d86+nWRpJVPwktU2s5P0MniJP97GrVEX2fkDx0nLiSkgTE9yCIvik\nhO9t020CgYEAxbRcW1jAB9cFeWKo4YxXAQv3DWc3PNYgNTAX8/GvS98DQC8LprtC\njMUPwaDQZBt1gMiPBYyYE8nqlIt6lHTJ8jZxQNIDYkbC8NNzPU26nj5UPfXGo5Io\nLYi4xiLXfbAZhTP7M2Bf3sJXAjENtPUh/AHexcBSHF3ZRPxxokxx9aQ=\n-----END RSA PRIVATE KEY-----\n'
CertComputeCreate
CertComputeCreate 用於建立具有證書的 Amphora 虛擬機器。
# file: /opt/rocky/octavia/octavia/controller/worker/tasks/compute_tasks.py
class CertComputeCreate(ComputeCreate):
def execute(self, amphora_id, server_pem,
build_type_priority=constants.LB_CREATE_NORMAL_PRIORITY,
server_group_id=None, ports=None):
"""Create an amphora
:returns: an amphora
"""
# load client certificate
with open(CONF.controller_worker.client_ca, 'r') as client_ca:
ca = client_ca.read()
config_drive_files = {
'/etc/octavia/certs/server.pem': server_pem,
'/etc/octavia/certs/client_ca.pem': ca}
return super(CertComputeCreate, self).execute(
amphora_id, config_drive_files=config_drive_files,
build_type_priority=build_type_priority,
server_group_id=server_group_id, ports=ports)
形參 server_pem 的實參是 GenerateServerPEMTask 的 return，也就是說 config_drive_files 包含了 CA 證書、Amphora 證書、Amphora 私鑰三者。最終通過 novaclient 呼叫 Nova 的 userdata 和 config drive 機制來完成 config_drive_files 檔案注入。
# file: /opt/rocky/octavia/octavia/compute/drivers/nova_driver.py
amphora = self.manager.create(
name=name, image=image_id, flavor=amphora_flavor,
key_name=key_name, security_groups=sec_groups,
nics=nics,
files=config_drive_files,
userdata=user_data,
config_drive=True,
scheduler_hints=server_group,
availability_zone=CONF.nova.availability_zone
)
登入到 Amphora Instance 可以檢視到這些檔案：
[email protected]:~$ ll /etc/octavia/certs/
total 16
drwxr-xr-x 2 root root 4096 Nov 5 02:36 ./
drwxr-xr-x 3 root root 4096 Nov 5 02:36 ../
-rw-rw---- 1 root root 1294 Nov 5 02:36 client_ca.pem
-rw-rw---- 1 root root 2964 Nov 5 02:36 server.pem
/etc/octavia/amphora-agent.conf 配置檔案的內容如下：
[DEFAULT]
debug = True
[haproxy_amphora]
base_cert_dir = /var/lib/octavia/certs
base_path = /var/lib/octavia
bind_host = ::
bind_port = 9443
haproxy_cmd = /usr/sbin/haproxy
respawn_count = 2
respawn_interval = 2
use_upstart = True
[health_manager]
controller_ip_port_list = 192.168.0.4:5555
heartbeat_interval = 10
heartbeat_key = insecure
[amphora_agent]
agent_server_ca = /etc/octavia/certs/client_ca.pem
agent_server_cert = /etc/octavia/certs/server.pem
agent_request_read_timeout = 120
amphora_id = a47ce718-1b3e-40a9-9fd9-a6ac5e1deba1
amphora_udp_driver = keepalived_lvs
[health_manager]section 描述了 amphora-agent 如何向 octavia-health-manager 上報監控狀態。[amphora_agent]section 中的 agent_server_ca 和 agent_server_cert 在啟動 amphora-agent 時候被載入啟用 SSL 通訊，後文再繼續展開。
Amphora Agent
amphora-agent 服務程序啟動指令碼：
# file: /usr/local/bin/amphora-agent
#!/opt/amphora-agent-venv/bin/python3
# -*- coding: utf-8 -*-
import re
import sys
from octavia.cmd.agent import main
if __name__ == '__main__':
sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])
sys.exit(main())
amphora-agent 服務程序的入口函式：
# file: /opt/rocky/octavia/octavia/cmd/agent.py
# start api server
def main():
# comment out to improve logging
service.prepare_service(sys.argv)
gmr.TextGuruMeditation.setup_autorun(version)
health_sender_proc = multiproc.Process(name='HM_sender',
target=health_daemon.run_sender,
args=(HM_SENDER_CMD_QUEUE,))
health_sender_proc.daemon = True
health_sender_proc.start()
# Initiate server class
server_instance = server.Server()
bind_ip_port = utils.ip_port_str(CONF.haproxy_amphora.bind_host,
CONF.haproxy_amphora.bind_port)
options = {
'bind': bind_ip_port,
'workers': 1,
'timeout': CONF.amphora_agent.agent_request_read_timeout,
'certfile': CONF.amphora_agent.agent_server_cert,
'ca_certs': CONF.amphora_agent.agent_server_ca,
'cert_reqs': True,
'preload_app': True,
'accesslog': '/var/log/amphora-agent.log',
'errorlog': '/var/log/amphora-agent.log',
'loglevel': 'debug',
}
AmphoraAgent(server_instance.app, options).run()
AmphoraAgent Application Class 繼承自 gunicorn.app.base.BaseApplication，通過 options 中的 CONF.amphora_agent.agent_server_cert 與 CONF.amphora_agent.agent_server_ca 載入 Amphora 的證書檔案路徑並啟用 SSL 通訊。
實際上 amphora-agent 使用的是 Flask 框架，在生成 Web Application 的過程中也完成了路由函式和檢視函式的初始化。
# file: /opt/rocky/octavia/octavia/amphorae/backends/agent/api_server/server.py
class Server(object):
def __init__(self):
self.app = flask.Flask(__name__)
self._osutils = osutils.BaseOS.get_os_util()
self._keepalived = keepalived.Keepalived()
self._listener = listener.Listener()
self._udp_listener = (udp_listener_base.UdpListenerApiServerBase.
get_server_driver())
self._plug = plug.Plug(self._osutils)
self._amphora_info = amphora_info.AmphoraInfo(self._osutils)
register_app_error_handler(self.app)
self.app.add_url_rule(rule=PATH_PREFIX +
'/listeners/<amphora_id>/<listener_id>/haproxy',
view_func=self.upload_haproxy_config,
methods=['PUT'])
...
AmphoraAPIClient
再回過頭來看看 AmphoraAPIClient 的實現，AmphoraAPIClient 在建立 REST 連線的 session 時會匯入了 create_certificates.sh 建立的 Server 證書，由 CONF.haproxy_amphora.client_cert 和 CONF.haproxy_amphora.server_ca 指定。
class AmphoraAPIClient(object):
def __init__(self):
super(AmphoraAPIClient, self).__init__()
self.secure = False
self.get = functools.partial(self.request, 'get')
self.post = functools.partial(self.request, 'post')
self.put = functools.partial(self.request, 'put')
self.delete = functools.partial(self.request, 'delete')
self.head = functools.partial(self.request, 'head')
self.start_listener = functools.partial(self._action,
consts.AMP_ACTION_START)
self.stop_listener = functools.partial(self._action,
consts.AMP_ACTION_STOP)
self.reload_listener = functools.partial(self._action,
consts.AMP_ACTION_RELOAD)
self.start_vrrp = functools.partial(self._vrrp_action,
consts.AMP_ACTION_START)
self.stop_vrrp = functools.partial(self._vrrp_action,
consts.AMP_ACTION_STOP)
self.reload_vrrp = functools.partial(self._vrrp_action,
consts.AMP_ACTION_RELOAD)
self.session = requests.Session()
self.session.cert = CONF.haproxy_amphora.client_cert
self.ssl_adapter = CustomHostNameCheckingAdapter()
self.session.mount('https://', self.ssl_adapter)
...
def request(self, method, amp, path='/', timeout_dict=None, **kwargs):
cfg_ha_amp = CONF.haproxy_amphora
if timeout_dict is None:
timeout_dict = {}
req_conn_timeout = timeout_dict.get(
consts.REQ_CONN_TIMEOUT, cfg_ha_amp.rest_request_conn_timeout)
req_read_timeout = timeout_dict.get(
consts.REQ_READ_TIMEOUT, cfg_ha_amp.rest_request_read_timeout)
conn_max_retries = timeout_dict.get(
consts.CONN_MAX_RETRIES, cfg_ha_amp.connection_max_retries)
conn_retry_interval = timeout_dict.get(
consts.CONN_RETRY_INTERVAL, cfg_ha_amp.connection_retry_interval)
LOG.debug("request url %s", path)
_request = getattr(self.session, method.lower())
_url = self._base_url(amp.lb_network_ip) + path
LOG.debug("request url %s", _url)
reqargs = {
'verify': CONF.haproxy_amphora.server_ca,
'url': _url,
'timeout': (req_conn_timeout, req_read_timeout), }
reqargs.update(kwargs)
headers = reqargs.setdefault('headers', {})
...
由於 Server 證書和 Amphora 證書都是由同一個 CA 中心簽發的，存在信任鏈關係，所以 AmphoraAPIClient 也就可以與 Amphora 進行 HTTPS 通訊了。
最後
一言以蔽之，Octavia 使用 CA 認證是為了保證 octavia-worker 和 Amphora 的通訊安全。我們需要注意的是相關配置專案的意義與證書的作用，通過分析程式碼實現的方式能夠對這個認證配置瞭解得更加深入。在 R 版本中，octavia-worker 和 Amphora 的通訊仍不能稱之為非常穩定，所以還是很有必要了解一下前因後果，以便候查。