| produce message | consume message| message's topic |
| send message time | delivery time, delivery rounds | message store location |
| whether the message was sent successfully | whether message was consumed successfully | message's key |
| send cost-time | consume cost-time| message's tag value |
## 2. Enable message trace in cluster deployment
### 2.1 Broker's configuration file
following by Broker's properties file configuration that enable message trace:
```
brokerClusterName=DefaultCluster
brokerName=broker-a
brokerId=0
deleteWhen=04
fileReservedTime=48
brokerRole=ASYNC_MASTER
flushDiskType=ASYNC_FLUSH
storePathRootDir=/data/rocketmq/rootdir-a-m
storePathCommitLog=/data/rocketmq/commitlog-a-m
autoCreateSubscriptionGroup=true
## if msg tracing is open,the flag will be true
traceTopicEnable=true
listenPort=10911
brokerIP1=XX.XX.XX.XX1
namesrvAddr=XX.XX.XX.XX:9876
```
### 2.2 Common mode
Each Broker node in RocketMQ cluster used for storing message trace data that client collected and sent. So, there is no requirements and limitations to the size of Broker node in RocketMQ cluster.
### 2.3 IO physical isolation mode
For huge amounts of message trace data scenario, we can select any one Broker node in RocketMQ cluster used for storing message trace data special, thus, common message data's IO are isolated from message trace data's IO in physical, not impact each other. In this mode, RocketMQ cluster must have at least two Broker nodes, the one that defined as storing message trace data.
### 2.4 Start Broker that enable message trace
`nohup sh mqbroker -c ../conf/2m-noslave/broker-a.properties &`
## 3. Save the definition of topic that with support message trace
RocketMQ's message trace feature supports two types of storage.
### 3.1 System level TraceTopic
Be default, message trace data is stored in system level TraceTopic(topic name: **RMQ_SYS_TRACE_TOPIC**). That topic will be created at startup of broker(As mentioned above, set **traceTopicEnable** to **true** in Broker's configuration).
### 3.2 User defined TraceTopic
If user don't want to store message trace data in system level TraceTopic, he can create user defined TraceTopic used for storing message trace data(that is, create common topic for storing message trace data). The following part will introduce how client SDK support user defined TraceTopic.
## 4. Client SDK demo with message trace feature
For business system adapting to use RocketMQ's message trace feature easily, in design phase, the author add a switch parameter(**enableMsgTrace**) for enable message trace; add a custom parameter(**customizedTraceTopic**) for user defined TraceTopic.
### 4.1 Enable message trace when sending messages
```
DefaultMQProducer producer = new DefaultMQProducer("ProducerGroupName",true);
