Uploaded image for project: 'Fabric'
  1. Fabric
  2. FAB-18400

Peers suddenly went down when using PKCS11 BCCSP

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Medium
    • Resolution: Won't Do
    • v2.2.0
    • None
    • None
    • Hide
      1. Build peer/orderer docker images with tag 'pkcs11' and softhsm
      (add 'apk add softhsm' in Dockerfile of peer/orderer)
      2. Configure and run a Fabric network with PCKS11 BCCSP and softhsm
      3. Create channel, deploy a chaincode, invoke chaincode
      4. Leave the network for 1~2 days
      Show
      1. Build peer/orderer docker images with tag 'pkcs11' and softhsm (add 'apk add softhsm' in Dockerfile of peer/orderer) 2. Configure and run a Fabric network with PCKS11 BCCSP and softhsm 3. Create channel, deploy a chaincode, invoke chaincode 4. Leave the network for 1~2 days

    Description

      I didn't anything on network for 20 hours or more, but peers went down with 'fatal error: unexpected signal during runtime execution' when signing through PKCS11 BCCSP. 
      It seems to occur during gossip communication between peers.

      Environment =>

      • OS : ubuntu 
      • HSM : softhsm 2.6.1 (in docker container)
      • Fabric : v2.2.0
      • Network : 1 orderer (etcdraft) and 2 peers (2 organizations each with 1 peer & anchor peer) 

       

      Error log is below. (some logs are too long, so I attached full log by file)

      ===============================================================

      1) softhsm

      user.err : BotanECDSAPrivateKey.cpp(223): Could not create the Botan public key
      user.err : BotanECDSA.cpp(93): Could not get the Botan private key
      user.err : fatal.cpp(56): A fatal exception occurred; exiting...

      ===============================================================

      2) peer0.org1.example.com
      [33m2021-01-15 20:54:32.498 UTC [gossip.gossip] SignMessage -> WARN 09b Failed signing message: P11: sign failed [pkcs11: 0x5: CKR_GENERAL_ERROR]
      github.com/hyperledger/fabric/gossip/gossip.(*discoverySecurityAdapter).SignMessage
      /go/src/github.com/hyperledger/fabric/gossip/gossip/gossip_impl.go:1047
      github.com/hyperledger/fabric/gossip/discovery.(*gossipDiscoveryImpl).createSignedAliveMessage
      /go/src/github.com/hyperledger/fabric/gossip/discovery/discovery_impl.go:800
      github.com/hyperledger/fabric/gossip/discovery.(*gossipDiscoveryImpl).createMembershipRequest
      /go/src/github.com/hyperledger/fabric/gossip/discovery/discovery_impl.go:659
      github.com/hyperledger/fabric/gossip/discovery.(*gossipDiscoveryImpl).InitiateSync
      /go/src/github.com/hyperledger/fabric/gossip/discovery/discovery_impl.go:233
      github.com/hyperledger/fabric/gossip/gossip.(*Node).syncDiscovery
      /go/src/github.com/hyperledger/fabric/gossip/gossip/gossip_impl.go:286
      runtime.goexit
      /usr/local/go/src/runtime/asm_amd64.s:1373
      2021-01-15 20:54:32.498 UTC [gossip.discovery] InitiateSync -> WARN 09c Failed creating membership request: Failed signing message
      github.com/hyperledger/fabric/gossip/discovery.(*gossipDiscoveryImpl).createSignedAliveMessage
      /go/src/github.com/hyperledger/fabric/gossip/discovery/discovery_impl.go:802
      github.com/hyperledger/fabric/gossip/discovery.(*gossipDiscoveryImpl).createMembershipRequest
      /go/src/github.com/hyperledger/fabric/gossip/discovery/discovery_impl.go:659
      github.com/hyperledger/fabric/gossip/discovery.(*gossipDiscoveryImpl).InitiateSync
      /go/src/github.com/hyperledger/fabric/gossip/discovery/discovery_impl.go:233
      github.com/hyperledger/fabric/gossip/gossip.(*Node).syncDiscovery
      /go/src/github.com/hyperledger/fabric/gossip/gossip/gossip_impl.go:286
      runtime.goexit
      /usr/local/go/src/runtime/asm_amd64.s:1373
      github.com/hyperledger/fabric/gossip/discovery.(*gossipDiscoveryImpl).createMembershipRequest
      /go/src/github.com/hyperledger/fabric/gossip/discovery/discovery_impl.go:661
      github.com/hyperledger/fabric/gossip/discovery.(*gossipDiscoveryImpl).InitiateSync
      /go/src/github.com/hyperledger/fabric/gossip/discovery/discovery_impl.go:233
      github.com/hyperledger/fabric/gossip/gossip.(*Node).syncDiscovery
      /go/src/github.com/hyperledger/fabric/gossip/gossip/gossip_impl.go:286
      runtime.goexit
      /usr/local/go/src/runtime/asm_amd64.s:1373
      github.com/hyperledger/fabric/gossip/discovery.(*gossipDiscoveryImpl).InitiateSync
      /go/src/github.com/hyperledger/fabric/gossip/discovery/discovery_impl.go:235
      github.com/hyperledger/fabric/gossip/gossip.(*Node).syncDiscovery
      /go/src/github.com/hyperledger/fabric/gossip/gossip/gossip_impl.go:286
      runtime.goexit
      /usr/local/go/src/runtime/asm_amd64.s:1373
      fatal error: unexpected signal during runtime execution
      [signal SIGSEGV: segmentation violation code=0x1 addr=0x7f97f61e0000 pc=0x7f981ddddccd]

      runtime stack:
      runtime.throw(0x192fd99, 0x2a)
      /usr/local/go/src/runtime/panic.go:1116 +0x72
      runtime.sigpanic()
      /usr/local/go/src/runtime/signal_unix.go:679 +0x46a
      ===============================================================

      3) peer0.org2.example.com
      fatal error: unexpected signal during runtime execution
      [signal SIGSEGV: segmentation violation code=0x1 addr=0x7fc8b0a8a040 pc=0x7fc8d8664ccd]

      runtime stack:
      runtime.throw(0x192fd99, 0x2a)
      /usr/local/go/src/runtime/panic.go:1116 +0x72
      runtime.sigpanic()
      /usr/local/go/src/runtime/signal_unix.go:679 +0x46a

      goroutine 60 [syscall]:
      runtime.cgocall(0x14fcbb0, 0xc002aa3770, 0x1)
      /usr/local/go/src/runtime/cgocall.go:133 +0x5b fp=0xc002aa3740 sp=0xc002aa3708 pc=0x9c492b
      github.com/miekg/pkcs11._Cfunc_SignInit(0x24de8c0, 0x1, 0xc003234b80, 0x2, 0x0)
      _cgo_gotypes.go:1397 +0x4e fp=0xc002aa3770 sp=0xc002aa3740 pc=0xdc007e
      github.com/miekg/pkcs11.(*Ctx).SignInit.func1(0xc000010f70, 0x1, 0xc003234b80, 0x2, 0x0)
      /go/src/github.com/hyperledger/fabric/vendor/github.com/miekg/pkcs11/pkcs11.go:1316 +0x99 fp=0xc002aa37b0 sp=0xc002aa3770 pc=0xdcccf9
      github.com/miekg/pkcs11.(*Ctx).SignInit(0xc000010f70, 0x1, 0xc002aa38c8, 0x1, 0x1, 0x2, 0x0, 0x0)
      /go/src/github.com/hyperledger/fabric/vendor/github.com/miekg/pkcs11/pkcs11.go:1316 +0xc7 fp=0xc002aa3820 sp=0xc002aa37b0 pc=0xdc5687
      github.com/hyperledger/fabric/bccsp/pkcs11.(*impl).signP11ECDSA(0xc000399020, 0xc0003c6940, 0x20, 0x20, 0xc003190d60, 0x20, 0x20, 0x0, 0x0, 0x0, ...)
      /go/src/github.com/hyperledger/fabric/bccsp/pkcs11/pkcs11.go:334 +0x286 fp=0xc002aa3900 sp=0xc002aa3820 pc=0xdd61b6
      github.com/hyperledger/fabric/bccsp/pkcs11.(*impl).signECDSA(0xc000399020, 0xc0003c6940, 0x20, 0x20, 0xc0003c6940, 0x20, 0x20, 0xc0003cc900, 0xc003190d60, 0x20, ...)
      /go/src/github.com/hyperledger/fabric/bccsp/pkcs11/ecdsa.go:18 +0x7a fp=0xc002aa3970 sp=0xc002aa3900 pc=0xdd02da
      github.com/hyperledger/fabric/bccsp/pkcs11.(*impl).Sign(0xc000399020, 0x1b1c060, 0xc00036c9c0, 0xc003190d60, 0x20, 0x20, 0x0, 0x0, 0xd9d02a, 0xc00014c050, ...)
      /go/src/github.com/hyperledger/fabric/bccsp/pkcs11/impl.go:182 +0x1e1 fp=0xc002aa3a50 sp=0xc002aa3970 pc=0xdd1f31
      github.com/hyperledger/fabric/bccsp/signer.(*bccspCryptoSigner).Sign(0xc0003ca3f0, 0x1ae7fe0, 0xc0001124b0, 0xc003190d60, 0x20, 0x20, 0x0, 0x0, 0x0, 0x0, ...)
      /go/src/github.com/hyperledger/fabric/bccsp/signer/signer.go:77 +0xed fp=0xc002aa3ae8 sp=0xc002aa3a50 pc=0xf4ea7d
      github.com/hyperledger/fabric/msp.(*signingidentity).Sign(0xc0003b9aa0, 0xc003190d40, 0x1d, 0x1d, 0xc003190d40, 0x0, 0x1d, 0xc002e54900, 0xc003190d40)
      /go/src/github.com/hyperledger/fabric/msp/identities.go:274 +0x38e fp=0xc002aa3bf8 sp=0xc002aa3ae8 pc=0xf5ad0e
      github.com/hyperledger/fabric/internal/peer/gossip.(*MSPMessageCryptoService).Sign(0xc0000d4040, 0xc003190d40, 0x1d, 0x1d, 0x7fc8b1900190, 0x1d, 0x1, 0xc002e54990, 0x30)
      /go/src/github.com/hyperledger/fabric/internal/peer/gossip/mcs.go:198 +0x56 fp=0xc002aa3c50 sp=0xc002aa3bf8 pc=0x14e7ca6
      github.com/hyperledger/fabric/gossip/gossip.(*discoverySecurityAdapter).SignMessage.func1(0xc003190d40, 0x1d, 0x1d, 0x1d, 0x1d, 0x0, 0x0, 0x46)
      /go/src/github.com/hyperledger/fabric/gossip/gossip/gossip_impl.go:1037 +0x55 fp=0xc002aa3ca8 sp=0xc002aa3c50 pc=0x132f2b5
      github.com/hyperledger/fabric/gossip/protoext.SignSecret(0xc002c2d9e0, 0xc002aa3da8, 0xc002e54990, 0x0, 0x0)
      /go/src/github.com/hyperledger/fabric/gossip/protoext/signing.go:31 +0x8a fp=0xc002aa3d28 sp=0xc002aa3ca8 pc=0x12dfd1a
      github.com/hyperledger/fabric/gossip/gossip.(*discoverySecurityAdapter).SignMessage(0xc0004ee100, 0xc002c2d980, 0xc000046072, 0x1b, 0x20)
      /go/src/github.com/hyperledger/fabric/gossip/gossip/gossip_impl.go:1054 +0x1cc fp=0xc002aa3dc8 sp=0xc002aa3d28 pc=0x132b69c
      github.com/hyperledger/fabric/gossip/discovery.(*gossipDiscoveryImpl).createSignedAliveMessage(0xc00013a420, 0x9c7c01, 0xc0000b50e0, 0x0, 0xc000112400)
      /go/src/github.com/hyperledger/fabric/gossip/discovery/discovery_impl.go:800 +0x56 fp=0xc002aa3e18 sp=0xc002aa3dc8 pc=0x12f2356
      github.com/hyperledger/fabric/gossip/discovery.(*gossipDiscoveryImpl).createMembershipRequest(0xc00013a420, 0xc0000e3701, 0xc003392510, 0x0, 0x1)
      /go/src/github.com/hyperledger/fabric/gossip/discovery/discovery_impl.go:659 +0x38 fp=0xc002aa3e80 sp=0xc002aa3e18 pc=0x12f05d8
      github.com/hyperledger/fabric/gossip/discovery.(*gossipDiscoveryImpl).InitiateSync(0xc00013a420, 0x3)
      /go/src/github.com/hyperledger/fabric/gossip/discovery/discovery_impl.go:233 +0x61 fp=0xc002aa3f70 sp=0xc002aa3e80 pc=0x12eacb1
      github.com/hyperledger/fabric/gossip/gossip.(*Node).syncDiscovery(0xc0001237c0)
      /go/src/github.com/hyperledger/fabric/gossip/gossip/gossip_impl.go:286 +0x167 fp=0xc002aa3fd8 sp=0xc002aa3f70 pc=0x13252f7
      runtime.goexit()
      /usr/local/go/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc002aa3fe0 sp=0xc002aa3fd8 pc=0xa29d41
      created by github.com/hyperledger/fabric/gossip/gossip.(*Node).start
      /go/src/github.com/hyperledger/fabric/gossip/gossip/gossip_impl.go:292 +0x43

       

      Attachments

        Activity

          People

            sykesm Matthew Sykes
            yjchang21 Yoojin Chang
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: