Uploaded image for project: 'Fabric'
  1. Fabric
  2. FAB-6260

ledgerstore uses re-entrant Rlock and deadlocks

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • High
    • Resolution: Done
    • v1.1.0
    • v1.1.0
    • fabric-ledger

    Description

      in core/ledger/ledgerstore/store.go there is a re-entrant RLock that might be while a Lock (write lock) comes in between.

      If that happens, the 2nd RLock waits for the Lock to finish, which never does because its waiting for the 1st RLock to be RUnlocked.

      func (s *Store) GetPvtDataAndBlockByNum(blockNum uint64, filter ledger.PvtNsCollFilter) (*ledger.BlockAndPvtData, error) {
      	s.rwlock.RLock()                   // 1st RLock
      	defer s.rwlock.RUnlock()
               ... 
      	if pvtdata, err = s.GetPvtDataByNum(blockNum, filter); err != nil {
      		return nil, err
      	}
            
      }
      
      func (s *Store) GetPvtDataByNum(blockNum uint64, filter ledger.PvtNsCollFilter) ([]*ledger.TxPvtData, error) {
      	s.rwlock.RLock()              // 2nd RLock
      	defer s.rwlock.RUnlock()
      
              ... 
      }
      

      This causes the following stack trace while a peer tries to catch up his fellow peer:

      goroutine 344 [semacquire, 2 minutes]:
      sync.runtime_Semacquire(0xc4226bd6a8)
              /opt/go/src/runtime/sema.go:56 +0x39
      sync.(*RWMutex).Lock(0xc4226bd6a0)
              /opt/go/src/sync/rwmutex.go:98 +0x6e
      github.com/hyperledger/fabric/core/ledger/ledgerstorage.(*Store).CommitWithPvtData(0xc4226bb7a0, 0xc4202eaec0, 0x0, 0x0)
              /opt/gopath/src/github.com/hyperledger/fabric/core/ledger/ledgerstorage/store.go:91 +0x5c
      github.com/hyperledger/fabric/core/ledger/kvledger.(*kvLedger).CommitWithPvtData(0xc4225c3600, 0xc4202eaec0, 0x0, 0x0)
              /opt/gopath/src/github.com/hyperledger/fabric/core/ledger/kvledger/kv_ledger.go:210 +0x28a
      github.com/hyperledger/fabric/core/ledger/ledgermgmt.(*closableLedger).CommitWithPvtData(0xc42270f440, 0xc4202eaec0, 0x0, 0x0)
              <autogenerated>:1 +0x47
      github.com/hyperledger/fabric/core/committer.(*LedgerCommitter).CommitWithPvtData(0xc421b40cc0, 0xc4202eaec0, 0xc421843d68, 0x1)
              /opt/gopath/src/github.com/hyperledger/fabric/core/committer/committer_impl.go:118 +0x6b
      github.com/hyperledger/fabric/gossip/privdata.(*coordinator).StoreBlock(0xc421b41b90, 0xc425df2b20, 0x0, 0x0, 0x0, 0x0, 0x2830)
              /opt/gopath/src/github.com/hyperledger/fabric/gossip/privdata/coordinator.go:142 +0xae7
      github.com/hyperledger/fabric/gossip/state.(*GossipStateProviderImpl).commitBlock(0xc421c2d000, 0xc425df2b20, 0x0, 0x0, 0x0, 0x0, 0x0)
              /opt/gopath/src/github.com/hyperledger/fabric/gossip/state/state.go:783 +0x7d
      github.com/hyperledger/fabric/gossip/state.(*GossipStateProviderImpl).deliverPayloads(0xc421c2d000)
              /opt/gopath/src/github.com/hyperledger/fabric/gossip/state/state.go:556 +0x411
      created by github.com/hyperledger/fabric/gossip/state.NewGossipStateProvider
              /opt/gopath/src/github.com/hyperledger/fabric/gossip/state/state.go:254 +0x6b9
      
      
      goroutine 346 [semacquire, 2 minutes]:
      sync.runtime_Semacquire(0xc4226bd6ac)
              /opt/go/src/runtime/sema.go:56 +0x39
      sync.(*RWMutex).RLock(0xc4226bd6a0)
              /opt/go/src/sync/rwmutex.go:50 +0x49
      github.com/hyperledger/fabric/core/ledger/ledgerstorage.(*Store).GetPvtDataByNum(0xc4226bb7a0, 0x82, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
              /opt/gopath/src/github.com/hyperledger/fabric/core/ledger/ledgerstorage/store.go:129 +0x66
      github.com/hyperledger/fabric/core/ledger/ledgerstorage.(*Store).GetPvtDataAndBlockByNum(0xc4226bb7a0, 0x82, 0x0, 0x0, 0x0, 0x0)
              /opt/gopath/src/github.com/hyperledger/fabric/core/ledger/ledgerstorage/store.go:119 +0xe4
      github.com/hyperledger/fabric/core/ledger/kvledger.(*kvLedger).GetPvtDataAndBlockByNum(0xc4225c3600, 0x82, 0x0, 0xc42189ac28, 0xc42c213c28, 0x411b58)
              /opt/gopath/src/github.com/hyperledger/fabric/core/ledger/kvledger/kv_ledger.go:233 +0x43
      github.com/hyperledger/fabric/core/ledger/ledgermgmt.(*closableLedger).GetPvtDataAndBlockByNum(0xc42270f440, 0x82, 0x0, 0x7b4168, 0xc420299650, 0x5)
              <autogenerated>:1 +0x51
      github.com/hyperledger/fabric/core/committer.(*LedgerCommitter).GetPvtDataAndBlockByNum(0xc421b40cc0, 0x82, 0xc421c2d580, 0xc42c213cb0, 0x7b4638)
              /opt/gopath/src/github.com/hyperledger/fabric/core/committer/committer_impl.go:131 +0x46
      github.com/hyperledger/fabric/gossip/privdata.(*coordinator).GetPvtDataAndBlockByNum(0xc421b41b90, 0x82, 0x3, 0x3, 0x96000, 0x0, 0x0, 0x0)
              /opt/gopath/src/github.com/hyperledger/fabric/gossip/privdata/coordinator.go:463 +0x58
      github.com/hyperledger/fabric/gossip/state.(*GossipStateProviderImpl).handleStateRequest(0xc421c2d000, 0x157e240, 0xc432974600)
              /opt/gopath/src/github.com/hyperledger/fabric/gossip/state/state.go:415 +0x42f
      github.com/hyperledger/fabric/gossip/state.(*GossipStateProviderImpl).processStateRequests(0xc421c2d000)
              /opt/gopath/src/github.com/hyperledger/fabric/gossip/state/state.go:372 +0x89
      created by github.com/hyperledger/fabric/gossip/state.NewGossipStateProvider
              /opt/gopath/src/github.com/hyperledger/fabric/gossip/state/state.go:258 +0x6fd
      
      

       

      The following demonstrates the root cause of the problem in a simple way.

      denyeart manish-sethi

       

      Attachments

        Activity

          People

            yacovm Yacov Manevich
            yacovm Yacov Manevich
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: