Uploaded image for project: 'Fabric'
  1. Fabric
  2. FAB-16610

Commit block to ledger hang when chaincode crash

    XMLWordPrintable

Details

    • Yes
    • Hide
      1. install and instantiate chaincode "demo" in channel(here "channel3" for example)
      2. call "exit" function use peer command line: `peer chaincode query -C channel3 -n demo -c '{"Args":["exit"]}'`
      3. use CTRL+C to interrupt query command
      4. send a new transaction through "demo" chaincode: `peer chaincode invoke -C channel3 -n demo -c '{"Args":["set", "key", "value"]}' --tls --cafile /root.pem`
      5. peer commit process hang until CORE_CHAINCODE_EXECUTETIMEOUT exceed
      Show
      1. install and instantiate chaincode "demo" in channel(here "channel3" for example) 2. call "exit" function use peer command line: `peer chaincode query -C channel3 -n demo -c '{"Args":["exit"]}'` 3. use CTRL+C to interrupt query command 4. send a new transaction through "demo" chaincode: `peer chaincode invoke -C channel3 -n demo -c '{"Args":["set", "key", "value"]}' --tls --cafile /root.pem` 5. peer commit process hang until CORE_CHAINCODE_EXECUTETIMEOUT exceed

    Description

      Peer create a TxSimulator(https://github.com/hyperledger/fabric/blob/release-1.4/core/endorser/endorser.go#L466) struct for each proposal which holds a read only lock in LockBasedTxMgr(https://github.com/hyperledger/fabric/blob/release-1.4/core/ledger/kvledger/txmgmt/txmgr/lockbasedtxmgr/lockbased_txmgr.go#L99) object. When chaincode crash and ChaincodeStream(https://github.com/hyperledger/fabric/blob/release-1.4/core/chaincode/handler.go#L393) ended, it hasn't release the TxSimulator object and it still holds the lock. When peer receive a new block and try to commit it to ledger, it hangs in there(https://github.com/hyperledger/fabric/blob/release-1.4/core/ledger/kvledger/txmgmt/txmgr/lockbasedtxmgr/lockbased_txmgr.go#L481) until proposal timeout. 

       

      Is it possible to return error response when chaincode stream failed and release the read only lock in LockBasedTxMgr?

       

      Peer logs:

      2019-09-16 02:10:36.002 UTC [chaincode] ProcessStream -> ERRO 4d5 handling chaincode support stream: rpc error: code = Canceled desc = context canceled
      receive failed
      github.com/hyperledger/fabric/core/chaincode.(*Handler).ProcessStream
      	/opt/gopath/src/github.com/hyperledger/fabric/core/chaincode/handler.go:427
      github.com/hyperledger/fabric/core/chaincode.(*ChaincodeSupport).HandleChaincodeStream
      	/opt/gopath/src/github.com/hyperledger/fabric/core/chaincode/chaincode_support.go:191
      github.com/hyperledger/fabric/core/chaincode.(*ChaincodeSupport).Register
      	/opt/gopath/src/github.com/hyperledger/fabric/core/chaincode/chaincode_support.go:196
      github.com/hyperledger/fabric/core/chaincode/accesscontrol.(*interceptor).Register
      	/opt/gopath/src/github.com/hyperledger/fabric/core/chaincode/accesscontrol/interceptor.go:57
      github.com/hyperledger/fabric/protos/peer._ChaincodeSupport_Register_Handler
      	/opt/gopath/src/github.com/hyperledger/fabric/protos/peer/chaincode_shim.pb.go:1069
      github.com/hyperledger/fabric/vendor/google.golang.org/grpc.(*Server).processStreamingRPC
      	/opt/gopath/src/github.com/hyperledger/fabric/vendor/google.golang.org/grpc/server.go:1124
      github.com/hyperledger/fabric/vendor/google.golang.org/grpc.(*Server).handleStream
      	/opt/gopath/src/github.com/hyperledger/fabric/vendor/google.golang.org/grpc/server.go:1212
      github.com/hyperledger/fabric/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1
      	/opt/gopath/src/github.com/hyperledger/fabric/vendor/google.golang.org/grpc/server.go:686
      runtime.goexit
      	/opt/go/src/runtime/asm_amd64.s:1333
      2019-09-16 02:10:41.198 UTC [gossip.privdata] StoreBlock -> INFO 4d6 [channel3] Received block [8] from buffer
      2019-09-16 02:10:41.199 UTC [committer.txvalidator] Validate -> INFO 4d7 [channel3] Validated block [8] in 0ms
      2019-09-16 02:15:35.991 UTC [endorser] SimulateProposal -> ERRO 4d8 [channel3][6499f87c] failed to invoke chaincode name:"demo" , error: timeout expired while executing transaction
      github.com/hyperledger/fabric/core/chaincode.(*Handler).Execute
      	/opt/gopath/src/github.com/hyperledger/fabric/core/chaincode/handler.go:1250
      github.com/hyperledger/fabric/core/chaincode.(*ChaincodeSupport).execute
      	/opt/gopath/src/github.com/hyperledger/fabric/core/chaincode/chaincode_support.go:313
      github.com/hyperledger/fabric/core/chaincode.(*ChaincodeSupport).Invoke
      	/opt/gopath/src/github.com/hyperledger/fabric/core/chaincode/chaincode_support.go:302
      github.com/hyperledger/fabric/core/chaincode.(*ChaincodeSupport).Execute
      	/opt/gopath/src/github.com/hyperledger/fabric/core/chaincode/chaincode_support.go:239
      github.com/hyperledger/fabric/core/endorser.(*SupportImpl).Execute
      	/opt/gopath/src/github.com/hyperledger/fabric/core/endorser/support.go:147
      github.com/hyperledger/fabric/core/endorser.(*Endorser).callChaincode
      	/opt/gopath/src/github.com/hyperledger/fabric/core/endorser/endorser.go:146
      github.com/hyperledger/fabric/core/endorser.(*Endorser).SimulateProposal
      	/opt/gopath/src/github.com/hyperledger/fabric/core/endorser/endorser.go:247
      github.com/hyperledger/fabric/core/endorser.(*Endorser).ProcessProposal
      	/opt/gopath/src/github.com/hyperledger/fabric/core/endorser/endorser.go:500
      github.com/hyperledger/fabric/core/handlers/auth/filter.(*expirationCheckFilter).ProcessProposal
      	/opt/gopath/src/github.com/hyperledger/fabric/core/handlers/auth/filter/expiration.go:61
      github.com/hyperledger/fabric/core/handlers/auth/filter.(*filter).ProcessProposal
      	/opt/gopath/src/github.com/hyperledger/fabric/core/handlers/auth/filter/filter.go:32
      github.com/hyperledger/fabric/protos/peer._Endorser_ProcessProposal_Handler.func1
      	/opt/gopath/src/github.com/hyperledger/fabric/protos/peer/peer.pb.go:169
      github.com/hyperledger/fabric/vendor/github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1
      	/opt/gopath/src/github.com/hyperledger/fabric/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:31
      github.com/hyperledger/fabric/core/comm.(*Throttle).UnaryServerIntercptor
      	/opt/gopath/src/github.com/hyperledger/fabric/core/comm/throttle.go:54
      github.com/hyperledger/fabric/core/comm.(*Throttle).UnaryServerIntercptor-fm
      	/opt/gopath/src/github.com/hyperledger/fabric/peer/node/start.go:228
      github.com/hyperledger/fabric/vendor/github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1
      	/opt/gopath/src/github.com/hyperledger/fabric/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:34
      github.com/hyperledger/fabric/common/grpclogging.UnaryServerInterceptor.func1
      	/opt/gopath/src/github.com/hyperledger/fabric/common/grpclogging/server.go:91
      github.com/hyperledger/fabric/vendor/github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1
      	/opt/gopath/src/github.com/hyperledger/fabric/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:34
      github.com/hyperledger/fabric/common/grpcmetrics.UnaryServerInterceptor.func1
      	/opt/gopath/src/github.com/hyperledger/fabric/common/grpcmetrics/interceptor.go:30
      github.com/hyperledger/fabric/vendor/github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1
      	/opt/gopath/src/github.com/hyperledger/fabric/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:39
      github.com/hyperledger/fabric/protos/peer._Endorser_ProcessProposal_Handler
      	/opt/gopath/src/github.com/hyperledger/fabric/protos/peer/peer.pb.go:171
      github.com/hyperledger/fabric/vendor/google.golang.org/grpc.(*Server).processUnaryRPC
      	/opt/gopath/src/github.com/hyperledger/fabric/vendor/google.golang.org/grpc/server.go:982
      github.com/hyperledger/fabric/vendor/google.golang.org/grpc.(*Server).handleStream
      	/opt/gopath/src/github.com/hyperledger/fabric/vendor/google.golang.org/grpc/server.go:1208
      github.com/hyperledger/fabric/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1
      	/opt/gopath/src/github.com/hyperledger/fabric/vendor/google.golang.org/grpc/server.go:686
      runtime.goexit
      	/opt/go/src/runtime/asm_amd64.s:1333
      error sending
      failed to execute transaction 6499f87cbd868ff02be7b2070fca922be7f9c0ef4421ff303501c4e7ee3aa227
      github.com/hyperledger/fabric/core/chaincode.processChaincodeExecutionResult
      	/opt/gopath/src/github.com/hyperledger/fabric/core/chaincode/chaincode_support.go:245
      github.com/hyperledger/fabric/core/chaincode.(*ChaincodeSupport).Execute
      	/opt/gopath/src/github.com/hyperledger/fabric/core/chaincode/chaincode_support.go:240
      github.com/hyperledger/fabric/core/endorser.(*SupportImpl).Execute
      	/opt/gopath/src/github.com/hyperledger/fabric/core/endorser/support.go:147
      github.com/hyperledger/fabric/core/endorser.(*Endorser).callChaincode
      	/opt/gopath/src/github.com/hyperledger/fabric/core/endorser/endorser.go:146
      github.com/hyperledger/fabric/core/endorser.(*Endorser).SimulateProposal
      	/opt/gopath/src/github.com/hyperledger/fabric/core/endorser/endorser.go:247
      github.com/hyperledger/fabric/core/endorser.(*Endorser).ProcessProposal
      	/opt/gopath/src/github.com/hyperledger/fabric/core/endorser/endorser.go:500
      github.com/hyperledger/fabric/core/handlers/auth/filter.(*expirationCheckFilter).ProcessProposal
      	/opt/gopath/src/github.com/hyperledger/fabric/core/handlers/auth/filter/expiration.go:61
      github.com/hyperledger/fabric/core/handlers/auth/filter.(*filter).ProcessProposal
      	/opt/gopath/src/github.com/hyperledger/fabric/core/handlers/auth/filter/filter.go:32
      github.com/hyperledger/fabric/protos/peer._Endorser_ProcessProposal_Handler.func1
      	/opt/gopath/src/github.com/hyperledger/fabric/protos/peer/peer.pb.go:169
      github.com/hyperledger/fabric/vendor/github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1
      	/opt/gopath/src/github.com/hyperledger/fabric/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:31
      github.com/hyperledger/fabric/core/comm.(*Throttle).UnaryServerIntercptor
      	/opt/gopath/src/github.com/hyperledger/fabric/core/comm/throttle.go:54
      github.com/hyperledger/fabric/core/comm.(*Throttle).UnaryServerIntercptor-fm
      	/opt/gopath/src/github.com/hyperledger/fabric/peer/node/start.go:228
      github.com/hyperledger/fabric/vendor/github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1
      	/opt/gopath/src/github.com/hyperledger/fabric/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:34
      github.com/hyperledger/fabric/common/grpclogging.UnaryServerInterceptor.func1
      	/opt/gopath/src/github.com/hyperledger/fabric/common/grpclogging/server.go:91
      github.com/hyperledger/fabric/vendor/github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1
      	/opt/gopath/src/github.com/hyperledger/fabric/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:34
      github.com/hyperledger/fabric/common/grpcmetrics.UnaryServerInterceptor.func1
      	/opt/gopath/src/github.com/hyperledger/fabric/common/grpcmetrics/interceptor.go:30
      github.com/hyperledger/fabric/vendor/github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1
      	/opt/gopath/src/github.com/hyperledger/fabric/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:39
      github.com/hyperledger/fabric/protos/peer._Endorser_ProcessProposal_Handler
      	/opt/gopath/src/github.com/hyperledger/fabric/protos/peer/peer.pb.go:171
      github.com/hyperledger/fabric/vendor/google.golang.org/grpc.(*Server).processUnaryRPC
      	/opt/gopath/src/github.com/hyperledger/fabric/vendor/google.golang.org/grpc/server.go:982
      github.com/hyperledger/fabric/vendor/google.golang.org/grpc.(*Server).handleStream
      	/opt/gopath/src/github.com/hyperledger/fabric/vendor/google.golang.org/grpc/server.go:1208
      github.com/hyperledger/fabric/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1
      	/opt/gopath/src/github.com/hyperledger/fabric/vendor/google.golang.org/grpc/server.go:686
      runtime.goexit
      	/opt/go/src/runtime/asm_amd64.s:1333
      2019-09-16 02:15:36.029 UTC [kvledger] CommitWithPvtData -> INFO 4d9 [channel3] Committed block [8] with 1 transaction(s) in 294829ms (state_validation=63ms block_and_pvtdata_commit=10ms state_commit=294753ms) commitHash=[c6a74215fa2e864e2b50b22872b1d082b1d81bbc724e34e951ff229dcaa18cdf]
      

       

      Attachments

        1. demo.go
          1 kB
          MingLiang Yin

        Activity

          People

            sykesm Matthew Sykes
            ymlbright MingLiang Yin
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: