We talk about the practices of China mobile on ceph object storage, the Multi-site datacenter architecture across 3 cities, the Billing System, new Features(bucket notification, request callback, object soft link) and The difficulties we encountered.