Uploaded image for project: 'Fabric'
  1. Fabric
  2. FAB-16880

Transaction proposal simulation continues after the corresponding gRPC request is canceled

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Unverified (View Workflow)
    • Priority: Medium
    • Resolution: Unresolved
    • Affects Version/s: v1.4.2
    • Fix Version/s: v2.x
    • Component/s: fabric-peer
    • Labels:
    • Steps to Reproduce:
      Hide
      Write a chaincode function which contains a long loop that does the following:

      - logs a message

      - does a GET_STATE or PUT_STATE

      - sleeps for 1 second

      Invoke this function via a client and kill the client program after a short time.

      The chaincode logs should show that the loop is still being executed despite the client disappearing.
      Show
      Write a chaincode function which contains a long loop that does the following: - logs a message - does a GET_STATE or PUT_STATE - sleeps for 1 second Invoke this function via a client and kill the client program after a short time. The chaincode logs should show that the loop is still being executed despite the client disappearing.

      Description

      If a request to endorse a transaction proposal has been canceled (e.g. due to the client crashing or another upstream timeout) the chaincode will continue the simulation even though there is no longer a client to receive the simulation result.

       

       

      The problem we have encountered is that a client will call our fabric-backed service with a request timeout that is too short for the transaction they are trying to have processed. After their timeout has elapsed they are canceling the request and retrying. The client's retries all time out because their timeout is just too short. But, each retry causes a new simulation to begin and we end up seeing simulations stacking up in our chaincode container which causes a spike in resource utilization (CPU and memory). In the worst case this can cause other, unrelated transactions to begin timing out as the CPU has become fully saturated.

       

      We are using fabric-sdk-go to issue transactions and tried connecting the context of the proposal endorsement to that of the incoming request using the package github.com/hyperledger/fabric-sdk-go/pkg/context (the WithParent function). When the incoming request is canceled the SDK immediately recognizes this, logs that the context was canceled, and returns.

       

       [fabsdk/fab] 2019/10/19 01:29:03 UTC - peer.(*peerEndorser).sendProposal -> ERRO process proposal failed [rpc error: code = Canceled desc = context canceled]
       [fabsdk/fab] 2019/10/19 01:29:03 UTC - peer.(*peerEndorser).sendProposal -> ERRO process proposal failed [rpc error: code = Canceled desc = context canceled]
      

       

      However, the chaincode container continues to process the canceled transaction and consume system resources.

        Attachments

          Activity

            People

            Assignee:
            Unassigned
            Reporter:
            bmatsuo Bryan Matsuo
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Dates

              Created:
              Updated:

                Git Integration