Details
-
Bug
-
Status: Closed
-
High
-
Resolution: Done
-
v1.1.0
Description
in core/ledger/ledgerstore/store.go there is a re-entrant RLock that might be while a Lock (write lock) comes in between.
If that happens, the 2nd RLock waits for the Lock to finish, which never does because its waiting for the 1st RLock to be RUnlocked.
func (s *Store) GetPvtDataAndBlockByNum(blockNum uint64, filter ledger.PvtNsCollFilter) (*ledger.BlockAndPvtData, error) {
s.rwlock.RLock() // 1st RLock
defer s.rwlock.RUnlock()
...
if pvtdata, err = s.GetPvtDataByNum(blockNum, filter); err != nil {
return nil, err
}
}
func (s *Store) GetPvtDataByNum(blockNum uint64, filter ledger.PvtNsCollFilter) ([]*ledger.TxPvtData, error) {
s.rwlock.RLock() // 2nd RLock
defer s.rwlock.RUnlock()
...
}
This causes the following stack trace while a peer tries to catch up his fellow peer:
goroutine 344 [semacquire, 2 minutes]: sync.runtime_Semacquire(0xc4226bd6a8) /opt/go/src/runtime/sema.go:56 +0x39 sync.(*RWMutex).Lock(0xc4226bd6a0) /opt/go/src/sync/rwmutex.go:98 +0x6e github.com/hyperledger/fabric/core/ledger/ledgerstorage.(*Store).CommitWithPvtData(0xc4226bb7a0, 0xc4202eaec0, 0x0, 0x0) /opt/gopath/src/github.com/hyperledger/fabric/core/ledger/ledgerstorage/store.go:91 +0x5c github.com/hyperledger/fabric/core/ledger/kvledger.(*kvLedger).CommitWithPvtData(0xc4225c3600, 0xc4202eaec0, 0x0, 0x0) /opt/gopath/src/github.com/hyperledger/fabric/core/ledger/kvledger/kv_ledger.go:210 +0x28a github.com/hyperledger/fabric/core/ledger/ledgermgmt.(*closableLedger).CommitWithPvtData(0xc42270f440, 0xc4202eaec0, 0x0, 0x0) <autogenerated>:1 +0x47 github.com/hyperledger/fabric/core/committer.(*LedgerCommitter).CommitWithPvtData(0xc421b40cc0, 0xc4202eaec0, 0xc421843d68, 0x1) /opt/gopath/src/github.com/hyperledger/fabric/core/committer/committer_impl.go:118 +0x6b github.com/hyperledger/fabric/gossip/privdata.(*coordinator).StoreBlock(0xc421b41b90, 0xc425df2b20, 0x0, 0x0, 0x0, 0x0, 0x2830) /opt/gopath/src/github.com/hyperledger/fabric/gossip/privdata/coordinator.go:142 +0xae7 github.com/hyperledger/fabric/gossip/state.(*GossipStateProviderImpl).commitBlock(0xc421c2d000, 0xc425df2b20, 0x0, 0x0, 0x0, 0x0, 0x0) /opt/gopath/src/github.com/hyperledger/fabric/gossip/state/state.go:783 +0x7d github.com/hyperledger/fabric/gossip/state.(*GossipStateProviderImpl).deliverPayloads(0xc421c2d000) /opt/gopath/src/github.com/hyperledger/fabric/gossip/state/state.go:556 +0x411 created by github.com/hyperledger/fabric/gossip/state.NewGossipStateProvider /opt/gopath/src/github.com/hyperledger/fabric/gossip/state/state.go:254 +0x6b9 goroutine 346 [semacquire, 2 minutes]: sync.runtime_Semacquire(0xc4226bd6ac) /opt/go/src/runtime/sema.go:56 +0x39 sync.(*RWMutex).RLock(0xc4226bd6a0) /opt/go/src/sync/rwmutex.go:50 +0x49 github.com/hyperledger/fabric/core/ledger/ledgerstorage.(*Store).GetPvtDataByNum(0xc4226bb7a0, 0x82, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /opt/gopath/src/github.com/hyperledger/fabric/core/ledger/ledgerstorage/store.go:129 +0x66 github.com/hyperledger/fabric/core/ledger/ledgerstorage.(*Store).GetPvtDataAndBlockByNum(0xc4226bb7a0, 0x82, 0x0, 0x0, 0x0, 0x0) /opt/gopath/src/github.com/hyperledger/fabric/core/ledger/ledgerstorage/store.go:119 +0xe4 github.com/hyperledger/fabric/core/ledger/kvledger.(*kvLedger).GetPvtDataAndBlockByNum(0xc4225c3600, 0x82, 0x0, 0xc42189ac28, 0xc42c213c28, 0x411b58) /opt/gopath/src/github.com/hyperledger/fabric/core/ledger/kvledger/kv_ledger.go:233 +0x43 github.com/hyperledger/fabric/core/ledger/ledgermgmt.(*closableLedger).GetPvtDataAndBlockByNum(0xc42270f440, 0x82, 0x0, 0x7b4168, 0xc420299650, 0x5) <autogenerated>:1 +0x51 github.com/hyperledger/fabric/core/committer.(*LedgerCommitter).GetPvtDataAndBlockByNum(0xc421b40cc0, 0x82, 0xc421c2d580, 0xc42c213cb0, 0x7b4638) /opt/gopath/src/github.com/hyperledger/fabric/core/committer/committer_impl.go:131 +0x46 github.com/hyperledger/fabric/gossip/privdata.(*coordinator).GetPvtDataAndBlockByNum(0xc421b41b90, 0x82, 0x3, 0x3, 0x96000, 0x0, 0x0, 0x0) /opt/gopath/src/github.com/hyperledger/fabric/gossip/privdata/coordinator.go:463 +0x58 github.com/hyperledger/fabric/gossip/state.(*GossipStateProviderImpl).handleStateRequest(0xc421c2d000, 0x157e240, 0xc432974600) /opt/gopath/src/github.com/hyperledger/fabric/gossip/state/state.go:415 +0x42f github.com/hyperledger/fabric/gossip/state.(*GossipStateProviderImpl).processStateRequests(0xc421c2d000) /opt/gopath/src/github.com/hyperledger/fabric/gossip/state/state.go:372 +0x89 created by github.com/hyperledger/fabric/gossip/state.NewGossipStateProvider /opt/gopath/src/github.com/hyperledger/fabric/gossip/state/state.go:258 +0x6fd
The following demonstrates the root cause of the problem in a simple way.