Uploaded image for project: 'Fabric'
  1. Fabric
  2. FAB-17343

Intermittent IT failure in raft: EndToEnd reconfiguration and onboarding when a single node cluster is expanded [It] is still possible to onboard the new cluster member and then another one with a different TLS root CA

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Unverified (View Workflow)
    • Priority: Highest
    • Resolution: Unresolved
    • Affects Version/s: v2.0.0
    • Fix Version/s: None
    • Component/s: fabric-orderer
    • Labels:
    • Steps to Reproduce:
      (Please add steps to reproduce)

      Description

      This was encountered during an integration test run against a PR:

      2020-01-09T14:56:38.7720372Z ------------------------------
      2020-01-09T14:56:38.7720596Z • Failure [84.520 seconds]
      2020-01-09T14:56:38.7720942Z EndToEnd reconfiguration and onboarding when a single node cluster is expanded [It] is still possible to onboard the new cluster member and then another one with a different TLS root CA 
      2020-01-09T14:56:38.7721215Z /home/vsts/work/1/go/src/github.com/hyperledger/fabric/integration/raft/config_test.go:242
      2020-01-09T14:56:38.7745324Z 
      2020-01-09T14:56:38.7745754Z   Timed out after 60.001s
      

      Log snippet:

      2020-01-09T14:56:38.7408591Z STEP: Adding the third orderer to the channel
      2020-01-09T14:56:38.7409186Z [d][peer-channel-fetch] starting peer channel fetch config --channelID systemchannel --orderer 127.0.0.1:25000 /tmp/e2e-etcfraft_reconfig730607004/getConfigBlock173755076/config_block.pb --tls --cafile /tmp/e2e-etcfraft_reconfig730607004/crypto/ca-certs.pem
      2020-01-09T14:56:38.7409513Z [e][peer-channel-fetch] 2020-01-09 14:55:36.681 UTC [channelCmd] InitCmdFactory -> INFO 001 Endorser and orderer connections initialized
      2020-01-09T14:56:38.7409825Z [e][peer-channel-fetch] 2020-01-09 14:55:36.683 UTC [cli.common] readBlock -> INFO 002 Received block: 1
      2020-01-09T14:56:38.7410136Z [e][peer-channel-fetch] 2020-01-09 14:55:36.683 UTC [channelCmd] fetch -> INFO 003 Retrieving last config block: 1
      2020-01-09T14:56:38.7410435Z [e][peer-channel-fetch] 2020-01-09 14:55:36.683 UTC [cli.common] readBlock -> INFO 004 Received block: 1
      2020-01-09T14:56:38.7411147Z [e][OrdererOrg.orderer] 2020-01-09 14:55:36.685 UTC [common.deliver] Handle -> WARN 068 Error reading from 127.0.0.1:40750: rpc error: code = Canceled desc = context canceled
      2020-01-09T14:56:38.7411617Z [e][OrdererOrg.orderer] 2020-01-09 14:55:36.685 UTC [comm.grpc.server] 1 -> INFO 069 streaming call completed grpc.service=orderer.AtomicBroadcast grpc.method=Deliver grpc.peer_address=127.0.0.1:40750 error="rpc error: code = Canceled desc = context canceled" grpc.code=Canceled grpc.call_duration=3.430067ms
      2020-01-09T14:56:38.7412066Z [d][peer-channel-fetch] starting peer channel fetch config --channelID systemchannel --orderer 127.0.0.1:25000 /tmp/e2e-etcfraft_reconfig730607004/currentConfigBlock813864598/config_block.pb --tls --cafile /tmp/e2e-etcfraft_reconfig730607004/crypto/ca-certs.pem
      2020-01-09T14:56:38.7412509Z [e][peer-channel-fetch] 2020-01-09 14:55:36.729 UTC [channelCmd] InitCmdFactory -> INFO 001 Endorser and orderer connections initialized
      2020-01-09T14:56:38.7412807Z [e][peer-channel-fetch] 2020-01-09 14:55:36.732 UTC [cli.common] readBlock -> INFO 002 Received block: 1
      2020-01-09T14:56:38.7413103Z [e][peer-channel-fetch] 2020-01-09 14:55:36.732 UTC [channelCmd] fetch -> INFO 003 Retrieving last config block: 1
      2020-01-09T14:56:38.7413415Z [e][peer-channel-fetch] 2020-01-09 14:55:36.734 UTC [cli.common] readBlock -> INFO 004 Received block: 1
      2020-01-09T14:56:38.7413763Z [e][OrdererOrg.orderer] 2020-01-09 14:55:36.735 UTC [common.deliver] Handle -> WARN 06a Error reading from 127.0.0.1:40752: rpc error: code = Canceled desc = context canceled
      2020-01-09T14:56:38.7414352Z [e][OrdererOrg.orderer] 2020-01-09 14:55:36.736 UTC [comm.grpc.server] 1 -> INFO 06b streaming call completed grpc.service=orderer.AtomicBroadcast grpc.method=Deliver grpc.peer_address=127.0.0.1:40752 error="rpc error: code = Canceled desc = context canceled" grpc.code=Canceled grpc.call_duration=5.814145ms
      2020-01-09T14:56:38.7414678Z [d][peer-channel-signconfigtx] starting peer channel signconfigtx --file /tmp/e2e-etcfraft_reconfig730607004/updateConfig879037779/update.pb
      2020-01-09T14:56:38.7415201Z [e][peer-channel-signconfigtx] 2020-01-09 14:55:36.779 UTC [channelCmd] InitCmdFactory -> INFO 001 Endorser and orderer connections initialized
      2020-01-09T14:56:38.7415814Z [d][peer-channel-update] starting peer channel update --channelID systemchannel --orderer 127.0.0.1:25000 --file /tmp/e2e-etcfraft_reconfig730607004/updateConfig879037779/update.pb --tls --cafile /tmp/e2e-etcfraft_reconfig730607004/crypto/ca-certs.pem
      2020-01-09T14:56:38.7416198Z [e][peer-channel-update] 2020-01-09 14:55:36.818 UTC [channelCmd] InitCmdFactory -> INFO 001 Endorser and orderer connections initialized
      2020-01-09T14:56:38.7417004Z [e][OrdererOrg.orderer] 2020-01-09 14:55:36.822 UTC [policies] SignatureSetToValidIdentities -> WARN 06c De-duplicating identity [OrdererMSP45aff807e067091bbab9c7e5f0958a6b4496306a589ea298bf099d361decc4e2] at index 1 in signature set
      2020-01-09T14:56:38.7417670Z [e][OrdererOrg.orderer] 2020-01-09 14:55:36.826 UTC [orderer.common.broadcast] ProcessMessage -> WARN 06d [channel: systemchannel] Rejecting broadcast of config message from 127.0.0.1:40754 because of error: error applying config update to existing channel 'systemchannel': consensus metadata update for channel config update is invalid: 0 out of 2 nodes are alive, configuration will result in quorum loss
      2020-01-09T14:56:38.7418172Z [e][OrdererOrg.orderer] 2020-01-09 14:55:36.827 UTC [comm.grpc.server] 1 -> INFO 06e streaming call completed grpc.service=orderer.AtomicBroadcast grpc.method=Broadcast grpc.peer_address=127.0.0.1:40754 grpc.code=OK grpc.call_duration=4.747355ms
      2020-01-09T14:56:38.7418696Z [e][peer-channel-update] Error: got unexpected status: BAD_REQUEST -- error applying config update to existing channel 'systemchannel': consensus metadata update for channel config update is invalid: 0 out of 2 nodes are alive, configuration will result in quorum loss
      

      Full logs:
      https://dev.azure.com/Hyperledger/0227bd2c-c4f6-478f-be00-ee519f115180/_apis/build/builds/4964/logs/61

        Attachments

        1. 79.txt
          337 kB
        2. flake_e2e.log
          327 kB
        3. raw_log_3-2-20.txt
          333 kB

          Issue Links

            Activity

              People

              Assignee:
              Unassigned
              Reporter:
              sykesm Matthew Sykes
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Dates

                Created:
                Updated:

                  Git Integration