Start time: 2018-09-04 12:20:00 End time: 2018-09-04 13:11:00 Duration: 0:51:00 Number of messages processed in one looper run: Node: 59679 samples, 14182.00 None, 0.00/0.24/55.00 min/avg/max, 2.32 stddev Client: 59679 samples, 48298.00 None, 0.00/0.81/100.00 min/avg/max, 8.80 stddev Seconds passed between looper runs: 59679 samples, 2963.88 None, 0.00/0.05/8.87 min/avg/max, 0.41 stddev Number of messages in one transport batch: 14940 samples, 1450355.00 None, 1.00/97.08/363.00 min/avg/max, 82.88 stddev Node message size, bytes: Outgoing: 14915 samples, 584925033.00 None, 2.00/39217.23/131071.00 min/avg/max, 32219.19 stddev Incoming: 14208 samples, 506844892.00 None, 2.00/35673.20/131072.00 min/avg/max, 29802.51 stddev Client message size, bytes: Outgoing: 95058 samples, 50535823.00 None, 81.00/531.63/1344.00 min/avg/max, 460.36 stddev Incoming: 48298 samples, 11812438.00 None, 158.00/244.57/247.00 min/avg/max, 13.81 stddev Number of requests in one 3PC batch: Created: 406 samples, 49760.00 None, 1.00/122.56/349.00 min/avg/max, 64.75 stddev Ordered: 382 samples, 46760.00 None, 1.00/122.41/349.00 min/avg/max, 65.46 stddev Time spent on write request processing, ms: 0.96/2.58/573.28 ms min/avg/max, 2.65 stddev Monitor throughput, TPS Master: 0.20/10.78/16.32 min/avg/max, 4.44 stddev Backup: 0.00/11.24/16.62 min/avg/max, 4.62 stddev Monitor latency, seconds Master: 1.31/68.74/109.90 min/avg/max, 28.56 stddev Backup: 1.36/67.17/110.97 min/avg/max, 27.93 stddev RAM info: Available, Mb: 4980.25/6920.64/7484.26 min/avg/max, 842.88 stddev Node RSS, Mb: 64.33/626.66/2547.51 min/avg/max, 837.90 stddev Node VMS, Mb: 423.75/1026.60/3060.11 min/avg/max, 929.10 stddev Additional statistics: Client incoming/outgoing: 0.51 messages, 0.23 traffic Node incoming/outgoing traffic: 0.87 Node/client traffic: 17.51 Node traffic per batch: 2858036.45 Node traffic per request: 23348.37 Profiling info: BACKUP_REQUEST_PROCESSING_TIME : 396680 samples, 0.54 seconds, 0.00/0.00/0.18 ms min/avg/max, 0.00 stddev REQUEST_PROCESSING_TIME : 49560 samples, 128.05 seconds, 0.96/2.58/573.28 ms min/avg/max, 2.65 stddev NODE_PROD_TIME : 59678 samples, 2380.28 seconds, 2.38/39.89/8869.06 ms min/avg/max, 406.49 stddev SERVICE_REPLICAS_TIME : 59679 samples, 297.30 seconds, 0.09/4.98/2191.84 ms min/avg/max, 59.12 stddev SERVICE_NODE_MSGS_TIME : 59679 samples, 1528.09 seconds, 0.54/25.61/3954.36 ms min/avg/max, 266.34 stddev SERVICE_CLIENT_MSGS_TIME : 59679 samples, 62.34 seconds, 0.12/1.04/149.49 ms min/avg/max, 9.93 stddev SERVICE_NODE_ACTIONS_TIME : 59678 samples, 1.81 seconds, 0.00/0.03/8.74 ms min/avg/max, 0.22 stddev SERVICE_LEDGER_MANAGER_TIME : 59678 samples, 0.17 seconds, 0.00/0.00/4.00 ms min/avg/max, 0.04 stddev SERVICE_VIEW_CHANGER_TIME : 59678 samples, 0.68 seconds, 0.00/0.01/8.61 ms min/avg/max, 0.06 stddev SERVICE_OBSERVABLE_TIME : 59678 samples, 0.40 seconds, 0.00/0.01/0.21 ms min/avg/max, 0.01 stddev SERVICE_OBSERVER_TIME : 59678 samples, 0.25 seconds, 0.00/0.00/0.15 ms min/avg/max, 0.00 stddev FLUSH_OUTBOXES_TIME : 59678 samples, 329.10 seconds, 0.01/5.51/4280.82 ms min/avg/max, 87.13 stddev SERVICE_NODE_LIFECYCLE_TIME : 59678 samples, 109.98 seconds, 0.02/1.84/23.12 ms min/avg/max, 0.28 stddev SERVICE_CLIENT_STACK_TIME : 59678 samples, 6.75 seconds, 0.08/0.11/10.23 ms min/avg/max, 0.38 stddev SERVICE_MONITOR_ACTIONS_TIME : 59678 samples, 37.42 seconds, 0.00/0.63/2311.74 ms min/avg/max, 31.91 stddev SERVICE_NODE_STACK_TIME : 59679 samples, 1243.42 seconds, 0.35/20.84/3319.53 ms min/avg/max, 213.81 stddev PROCESS_NODE_INBOX_TIME : 59679 samples, 280.90 seconds, 0.00/4.71/726.14 ms min/avg/max, 52.62 stddev SEND_TO_REPLICA_TIME : 167401 samples, 2.70 seconds, 0.01/0.02/0.44 ms min/avg/max, 0.01 stddev NODE_CHECK_PERFORMANCE_TIME : 252 samples, 0.04 seconds, 0.02/0.14/0.77 ms min/avg/max, 0.22 stddev NODE_CHECK_NODE_REQUEST_SPIKE : 47 samples, 0.00 seconds, 0.02/0.03/0.05 ms min/avg/max, 0.01 stddev UNPACK_BATCH_TIME : 12651 samples, 1122.08 seconds, 0.19/88.69/965.27 ms min/avg/max, 74.09 stddev VERIFY_SIGNATURE_TIME : 1267090 samples, 714.28 seconds, 0.46/0.56/5.42 ms min/avg/max, 0.14 stddev SERVICE_REPLICAS_OUTBOX_TIME : 59679 samples, 66.88 seconds, 0.00/1.12/531.89 ms min/avg/max, 15.44 stddev NODE_SEND_TIME : 61015 samples, 4.28 seconds, 0.03/0.07/0.79 ms min/avg/max, 0.02 stddev NODE_SEND_REJECT_TIME : no samples VALIDATE_NODE_MSG_TIME : 1400752 samples, 1045.44 seconds, 0.06/0.75/870.97 ms min/avg/max, 2.71 stddev INT_VALIDATE_NODE_MSG_TIME : 1400752 samples, 187.21 seconds, 0.02/0.13/870.12 ms min/avg/max, 2.69 stddev PROCESS_ORDERED_TIME : 3440 samples, 65.83 seconds, 0.04/19.14/395.30 ms min/avg/max, 43.10 stddev MONITOR_REQUEST_ORDERED_TIME : 3440 samples, 21.35 seconds, 0.02/6.21/29.91 ms min/avg/max, 5.39 stddev EXECUTE_BATCH_TIME : 381 samples, 43.84 seconds, 2.55/115.06/373.99 ms min/avg/max, 63.78 stddev SERVICE_REPLICA_QUEUES_TIME : 59679 samples, 191.58 seconds, 0.01/3.21/1761.32 ms min/avg/max, 41.27 stddev SERVICE_BACKUP_REPLICAS_QUEUES_TIME : 477432 samples, 34.45 seconds, 0.00/0.07/634.11 ms min/avg/max, 1.43 stddev PROCESS_PREPREPARE_TIME : 405 samples, 133.98 seconds, 3.58/330.82/1182.61 ms min/avg/max, 183.19 stddev PROCESS_PREPARE_TIME : 8485 samples, 1.26 seconds, 0.06/0.15/4.24 ms min/avg/max, 0.28 stddev PROCESS_COMMIT_TIME : 6140 samples, 53.13 seconds, 6.88/8.65/21.50 ms min/avg/max, 2.06 stddev PROCESS_CHECKPOINT_TIME : 71 samples, 0.80 seconds, 0.11/11.26/693.27 ms min/avg/max, 82.46 stddev SEND_PREPREPARE_TIME : no samples SEND_PREPARE_TIME : 405 samples, 0.12 seconds, 0.21/0.30/0.53 ms min/avg/max, 0.06 stddev SEND_COMMIT_TIME : 394 samples, 0.50 seconds, 0.98/1.26/4.13 ms min/avg/max, 0.33 stddev SEND_CHECKPOINT_TIME : 3 samples, 0.00 seconds, 0.51/0.62/0.81 ms min/avg/max, 0.16 stddev CREATE_3PC_BATCH_TIME : no samples ORDER_3PC_BATCH_TIME : 381 samples, 2.04 seconds, 4.04/5.35/8.86 ms min/avg/max, 1.04 stddev BACKUP_PROCESS_PREPREPARE_TIME : 2853 samples, 5.85 seconds, 0.35/2.05/7.08 ms min/avg/max, 0.98 stddev BACKUP_PROCESS_PREPARE_TIME : 69521 samples, 5.76 seconds, 0.05/0.08/4.12 ms min/avg/max, 0.03 stddev BACKUP_PROCESS_COMMIT_TIME : 49237 samples, 6.36 seconds, 0.05/0.13/15.26 ms min/avg/max, 0.25 stddev BACKUP_PROCESS_CHECKPOINT_TIME : 568 samples, 2.34 seconds, 0.10/4.12/612.04 ms min/avg/max, 34.43 stddev BACKUP_SEND_PREPREPARE_TIME : 419 samples, 0.04 seconds, 0.04/0.09/0.29 ms min/avg/max, 0.04 stddev BACKUP_SEND_PREPARE_TIME : 2853 samples, 0.37 seconds, 0.09/0.13/0.39 ms min/avg/max, 0.03 stddev BACKUP_SEND_COMMIT_TIME : 3165 samples, 0.31 seconds, 0.08/0.10/0.29 ms min/avg/max, 0.02 stddev BACKUP_SEND_CHECKPOINT_TIME : 24 samples, 0.01 seconds, 0.29/0.35/0.60 ms min/avg/max, 0.08 stddev BACKUP_CREATE_3PC_BATCH_TIME : 419 samples, 0.68 seconds, 0.18/1.63/6.31 ms min/avg/max, 0.95 stddev BACKUP_ORDER_3PC_BATCH_TIME : 3059 samples, 2.77 seconds, 0.38/0.91/15.12 ms min/avg/max, 0.41 stddev PROCESS_PROPAGATE_TIME : 1219990 samples, 236.55 seconds, 0.14/0.19/3.08 ms min/avg/max, 0.07 stddev PROCESS_MESSAGE_REQ_TIME : 303 samples, 0.09 seconds, 0.24/0.29/0.53 ms min/avg/max, 0.04 stddev PROCESS_MESSAGE_REP_TIME : 267 samples, 0.15 seconds, 0.38/0.56/7.39 ms min/avg/max, 0.69 stddev PROCESS_LEDGER_STATUS_TIME : 1499 samples, 1.06 seconds, 0.27/0.70/7.27 ms min/avg/max, 0.36 stddev PROCESS_CONSISTENCY_PROOF_TIME : no samples PROCESS_CATCHUP_REQ_TIME : no samples PROCESS_CATCHUP_REP_TIME : no samples PROCESS_REQUEST_TIME : 47100 samples, 10.59 seconds, 0.17/0.22/2.72 ms min/avg/max, 0.08 stddev SEND_PROPAGATE_TIME : 53136 samples, 11.79 seconds, 0.18/0.22/2.47 ms min/avg/max, 0.05 stddev SEND_MESSAGE_REQ_TIME : 240 samples, 0.02 seconds, 0.07/0.08/0.12 ms min/avg/max, 0.01 stddev SEND_MESSAGE_REP_TIME : 303 samples, 0.04 seconds, 0.10/0.12/0.23 ms min/avg/max, 0.02 stddev BLS_VALIDATE_PREPREPARE_TIME : 3258 samples, 3.91 seconds, 0.00/1.20/17.98 ms min/avg/max, 3.68 stddev BLS_VALIDATE_COMMIT_TIME : 55377 samples, 50.58 seconds, 0.00/0.91/14.12 ms min/avg/max, 2.63 stddev BLS_UPDATE_PREPREPARE_TIME : 419 samples, 0.00 seconds, 0.00/0.00/0.01 ms min/avg/max, 0.00 stddev BLS_UPDATE_COMMIT_TIME : 394 samples, 0.43 seconds, 0.86/1.09/3.88 ms min/avg/max, 0.31 stddev DESERIALIZE_DURING_UNPACK_TIME : no samples