Create a keyring for the gateway：
sudo ceph-authtool --create-keyring /etc/ceph/ceph.client.radosgw.keyring
sudo chmod +r /etc/ceph/ceph.client.radosgw.keyring


2. Generate a Ceph Object Gateway user name and key for each instance
For exemplary purposes, we will use the name gateway after client.radosgw:
sudo ceph-authtool /etc/ceph/ceph.client.radosgw.keyring -n client.radosgw.gateway --gen-key
 

Add capabilities to the key. See Configuration Reference - Pools for details on the effect of write permissions for the monitor and creating pools.
sudo ceph-authtool -n client.radosgw.gateway --cap osd 'allow rwx' --cap mon 'allow rwx' /etc/ceph/ceph.client.radosgw.keyring

3. 將key新增到ceph叢集中 
Once you have created a keyring and key to enable the Ceph Object Gateway with access to the Ceph Storage Cluster, add the key to your Ceph Storage Cluster. For example:
sudo ceph -k /etc/ceph/ceph.client.admin.keyring auth add client.radosgw.gateway -i /etc/ceph/ceph.client.radosgw.keyring
注意：實際在我的環境上路徑是這個：
sudo ceph -k 
 
/etc/ceph/keyring.client.admin auth add client.radosgw.gateway -i /etc/ceph/ceph.client.radosgw.keyring

4. 如果想在多個節點都裝客戶端
Distribute the keyring to the node with the gateway instance：
sudo scp /etc/ceph/ceph.client.radosgw.keyring  [email protected]{hostname}:/home/ceph
ssh {hostname}
sudo mv ceph.client.radosgw.keyring /etc/ceph/ceph.client.radosgw.keyring

5. 建立pool
Ceph Object Gateways require Ceph Storage Cluster pools to store specific gateway data. 在有些情況下有些pool是可以在執行過程中預設建立的，但是有些可能建立不成功，
所以建議手工建立吧：
ceph osd pool create .rgw 128 128
ceph osd pool create .rgw.root 128 128
ceph osd pool create .rgw.control 128 128
ceph osd pool create .rgw.gc 128 128
ceph osd pool create .rgw.buckets 128 128
ceph osd pool create .rgw.buckets.index 128 128
ceph osd pool create .log 128 128
ceph osd pool create .intent-log 128 128
ceph osd pool create .usage 128 128
ceph osd pool create .users 128 128
ceph osd pool create .users.email 128 128
ceph osd pool create .users.swift 128 128
ceph osd pool create .users.uid 128 128

6. 新增一個gateway配置到ceph.conf

[client.radosgw.{instance-name}]
host = {host-name}
keyring = /etc/ceph/ceph.client.radosgw.keyring
rgw socket path = /var/run/ceph/ceph.radosgw.{instance-name}.fastcgi.sock
log file = /var/log/ceph/client.radosgw.{instance-name}.log

這裡的{instance-name}是gateway：

[client.radosgw.gateway]
host = ceph-gateway
keyring = /etc/ceph/ceph.client.radosgw.keyring
rgw socket path = /var/run/ceph/ceph.radosgw.gateway.fastcgi.sock
log file = /var/log/ceph/client.radosgw.gateway.log

以下配置為非必須：

CONFIGURING PRINT CONTINUE
On CentOS/RHEL distributions, turn off print continue. If you have it set to true, you may encounter problems with PUT operations.

rgw print continue = false

CONFIGURING OPERATIONS LOGGING
In early releases of Ceph (v0.66 and earlier), the Ceph Object Gateway will log every successful operation in the Ceph Object Gateway backend by default. This means that every request, whether it is a read request or a write request will generate a gateway operation that writes data. This does not come without cost, and may affect overall performance. Turning off logging completely can be done by adding the following config option to the Ceph configuration file:

rgw enable ops log = false

Another way to reduce the logging load is to send operations logging data to a UNIX domain socket, instead of writing it to the Ceph Object Gateway backend:
rgw ops log rados = false
rgw enable ops log = true
rgw ops log socket path = <path to socket>

When specifying a UNIX domain socket, it is also possible to specify the maximum amount of memory that will be used to keep the data backlog:
rgw ops log data backlog = <size in bytes>

Any backlogged data in excess to the specified size will be lost, so the socket needs to be read constantly.


7. 使能S3的子域名呼叫，這一步不確定是不是必須
ENABLING SUBDOMAIN S3 CALLS
To use a Ceph Object Gateway with subdomain S3 calls (e.g., http://bucketname.hostname), you must add the Ceph Object Gateway DNS name under the [client.radosgw.gateway] section of your Ceph configuration file:
還需要在/etc/ceph.conf的gateway配置中新增：
[client.radosgw.gateway]
 ...
 rgw dns name = {hostname}

You should also consider installing a DNS server such as Dnsmasq on your client machine(s) when using http://{bucketname}.{hostname} syntax. The dnsmasq.conf file should include the following settings:

安裝dnsmasq：
$ sudo apt-get install dnsmasq

You should also consider installing a DNS server such as Dnsmasq on your client machine(s) when using http://{bucketname}.{hostname} syntax. The dnsmasq.conf file should include the following settings:
address=/{hostname}/{host-ip-address}
listen-address={client-loopback-ip}

我這裡的配置：
$ vim /etc/dnsmasq.conf中新增：
address=/node29/10.2.0.29
listen-address=127.0.0.1

Then, add the {client-loopback-ip} IP address as the first DNS nameserver on client the machine(s).
See Add Wildcard to DNS for details.

sudo /etc/init.d/dnsmasq restart

8. 增加一個ceph物件閘道器指令碼
ADD A CEPH OBJECT GATEWAY SCRIPT
Add a s3gw.fcgi file (use the same name referenced in the first line of rgw.conf). For Debian/Ubuntu distributions, save the file to the /var/www directory. For CentOS/RHEL distributions, save the file to the /var/www/html directory. Assuming a cluster named ceph (default), and the user created in previous steps, the contents of the file should include:
增加一個/var/www/s3gw.fcgi配置檔案（use the same name referenced in the first line of rgw.conf）
$vim /var/www/s3gw.fcgi
#!/bin/sh
exec /usr/bin/radosgw -c /etc/ceph/ceph.conf -n client.radosgw.gateway

對s3gw.fcgi增加可執行許可權：
sudo chmod +x s3gw.fcgi

On some distributions, you must also change the ownership to apache.
sudo chown apache:apache s3gw.fcgi

注意：ubuntu 預設的apache使用者是www-data
sudo chown www-data:www-data s3gw.fcgi

建立資料目錄：
CREATE DATA DIRECTORY
Deployment scripts may not create the default Ceph Object Gateway data directory. Create data directories for each instance of a radosgw daemon (if you haven’t done so already). The host variables in the Ceph configuration file determine which host runs each instance of a radosgw daemon. The typical form specifies the radosgw daemon, the cluster name and the daemon ID.

sudo mkdir -p /var/lib/ceph/radosgw/{$cluster}-{$id}

Using the exemplary ceph.conf settings above, you would execute the following:
sudo mkdir -p /var/lib/ceph/radosgw/ceph-radosgw.gateway

建立一個閘道器配置檔案：
CREATE A GATEWAY CONFIGURATION

On the host where you installed the Ceph Object Gateway, create an rgw.conf file. For Debian/Ubuntu systems, place the file in the /etc/apache2/sites-available directory. For CentOS/RHEL systems, place the file in the /etc/httpd/conf.d directory.
We recommend deploying FastCGI as an external server, because allowing Apache to manage FastCGI sometimes introduces high latency. To manage FastCGI as an external server, use the FastCgiExternalServer directive. See FastCgiExternalServer for details on this directive. See Module mod_fastcgi for general details. See Apache Virtual Host documentation for details on <VirtualHost> format and settings. See <IfModule> Directive for additional details.
Ceph Object Gateway requires a rewrite rule for the Amazon S3-compatible interface. It’s required for passing in the HTTP_AUTHORIZATION env for S3, which is filtered out by Apache. The rewrite rule is not necessary for the OpenStack Swift-compatible interface.
You should configure Apache to allow encoded slashes, provide paths for log files and to turn off server signatures. See below for an exemplary embodiment of a gateway configuration for Debian/Ubuntu and CentOS/RHEL.

$vim /etc/apache2/sites-available/rgw.conf

FastCgiExternalServer /var/www/s3gw.fcgi -socket /var/run/ceph/ceph.radosgw.gateway.fastcgi.sock

<VirtualHost *:80>

 ServerName node29
 ServerAlias *.node29
 ServerAdmin [email protected]
 DocumentRoot /var/www
 RewriteEngine On
 RewriteRule ^/(.*) /s3gw.fcgi?%{QUERY_STRING} [E=HTTP_AUTHORIZATION:%{HTTP:Authorization},L]

 <IfModule mod_fastcgi.c>
 <Directory /var/www>
 Options +ExecCGI
 AllowOverride All
 SetHandler fastcgi-script
 Order allow,deny
 Allow from all
 AuthBasicAuthoritative Off
 </Directory>
 </IfModule>

 AllowEncodedSlashes On
 ErrorLog /var/log/apache2/error.log
 CustomLog /var/log/apache2/access.log combined
 ServerSignature Off
</VirtualHost>

9. 使rgw.conf配置生效
For Debian/Ubuntu distributions, enable the site for rgw.conf.
sudo a2ensite rgw.conf
Then, disable the default site.
sudo a2dissite default

RESTART SERVICES AND START THE GATEWAY
To ensure that all components have reloaded their configurations, we recommend restarting your ceph and apache services. Then, start up the radosgw service.

RESTART APACHE
On Debian/Ubuntu systems, use apache2. For example:

sudo service apache2 restart

START THE GATEWAY
On Debian/Ubuntu systems, use radosgw. For example:

sudo /etc/init.d/radosgw start

VERIFY THE RUNTIME
Once the service is up and running, you can make an anonymous GET request to see if the gateway returns a response. A simple HTTP request to the domain name should return the following:

可以用curl來執行：
curl node29
成功則顯示以下相應，同時在日誌中會有記錄：

<ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
 <Owner>
 <ID>anonymous</ID>
 <DisplayName/>
 </Owner>
 <Buckets/>
</ListAllMyBucketsResult>

配置一個使用者，配置s3cmd進行S3介面測試

CREATE A USER
To create a user (S3 interface), execute the following:
radosgw-admin user create --uid={username} --display-name="{display-name}" [--email={email}]
For example:
radosgw-admin user create --uid=cephtest --display-name="ceph test" [email protected]
顯示：
{ "user_id": "cephtest",
 "display_name": "ceph test",
 "email": "[email protected]",
 "suspended": 0,
 "max_buckets": 1000,
 "auid": 0,
 "subusers": [],
 "keys": [
 { "user": "cephtest",
 "access_key": "8JGTP1714JRRAPRDLBI4",
 "secret_key": "i4O+yJMNii87ruLMwcIIYbjLmKIUaoSO1svQOoB9"}],
 "swift_keys": [],
 "caps": [],
 "op_mask": "read, write, delete",
 "default_placement": "",
 "placement_tags": [],
 "bucket_quota": { "enabled": false,
 "max_size_kb": -1,
 "max_objects": -1},
 "user_quota": { "enabled": false,
 "max_size_kb": -1,
 "max_objects": -1},
 "temp_url_keys": []}

注意：如果生成的key中有"\",最後把這個user刪掉，再重新生成直到不含"\"：
radosgw-admin user rm --uid=cephtest

Important：Check the key output. Sometimes radosgw-admin generates a JSON escape (\) character, and some clients do not know how to handle JSON escape characters. Remedies include removing the JSON escape character (\), encapsulating the string in quotes, regenerating the key and ensuring that it does not have a JSON escape character or specify the key and secret manually.

安裝和配置s3cmd
apt-get install s3cmd

進行~/.s3cfg配置, 將之前radosgw-admin建立的user的access_key和secret_key，實際的host地址進行賦值：

$vim ~/.s3cfg 

[default]
access_key = PPA789F4W5ANH3COC51O
bucket_location = US
cloudfront_host = cloudfront.amazonaws.com
cloudfront_resource = /2010-07-15/distribution
default_mime_type = binary/octet-stream
delete_removed = False
dry_run = False
encoding = UTF-8
encrypt = False
follow_symlinks = False
force = False
get_continue = False
gpg_command = /usr/bin/gpg
gpg_decrypt = %(gpg_command)s -d --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s
gpg_encrypt = %(gpg_command)s -c --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s
gpg_passphrase =
guess_mime_type = True
host_base = node29
host_bucket = %(bucket)s.node29
human_readable_sizes = False
list_md5 = False
log_target_prefix =
preserve_attrs = True
progress_meter = True
proxy_host =
proxy_port = 0
recursive = False
recv_chunk = 4096
reduced_redundancy = False
secret_key = kqHLxnI14WqSl0Eh5akr73evrqmFZjfxYxUmG04h
send_chunk = 4096
simpledb_host = sdb.amazonaws.com
skip_existing = False
socket_timeout = 10
urlencoding_mode = normal
use_https = False
verbosity = WARNING

測試是否配置成功：
s3cmd --configure

一路選確定，因為我們已經配置過了，直到最後一步Test選y：
Test access with supplied credentials? [Y/n] y
Please wait...
Success. Your access key and secret key worked fine :-)

Now verifying that encryption works...
Not configured. Never mind.

大功告成！

後記
簡單的測試命令：
幫助命令：

s3cmd --help

建立一個名為rgw的bucket：
s3cmd mb s3://rgw

將tempfile寫入到rgw這個bucket中：
s3cmd put tempfile s3://rgw

列出rgw這個bucket中的檔案列表：
s3cmd ls s3://rgw

讀取bucket中的tempfile檔案到本地：
s3cmd get s3://rgw/tempfile tempfile_new