indy_sign_and_submit_request hangs when NYM is created as a trust anchor in the same transaction

indy_sign_and_submit_request hangs when a NYM is created as a trust anchor in the same transaction.

I am developing an AWS Lambda (and associated webapp) that will allow potential trust anchors to add themselves as trust anchors on the Sovrin Test Network (STN). This would allow developers/customers to play at will as trust anchors on the STN w/o waiting on Evernym or Sovrin employees/stewards to add them as trust anchors.

I developed and tested the Lambda from my workstation and only experienced the "hang" a couple of times. In other words, the "hang" is inconsistent while working on my laptop connected to the internet via wi-fi with CenturyLink as our Internet Service Provider (ISP).

The "hang" is consistent when invoking the same code as an AWS Lambda.

The reason for this level of environment detail is that devin-fisher, krw910, and I believe that AWS Lambda is dispatching requests much faster to the STN than the typical development environment would. In other words, AWS likely has a MUCH faster network and thus suggests a race condition exists.

What we think is happening is that a sufficient number of validator nodes reach consensus and all other validator nodes "catchup" before processing their respective request. When the validator nodes that did not participate in consensus, process their respective request, the NYM appears to already exist on the ledger and the steward does not have permission to complete the request.

The following is an example response:

INFO|indy::services::pool | src/services/pool/mod.rs:557 | RemoteNode::recv_msg brazil
{ "reason": "client request invalid: InvalidClientRequest('HfXThVwhJB4o1Q1Fjr4yrC is neither Trustee nor owner of Nxnm242JjPJNEnvAuhDZiX',)", "reqId": 1509631735685633500, "op": "REQNACK", "identifier": "HfXThVwhJB4o1Q1Fjr4yrC" }
 
I have attached logs from both my workstation and the AWS Lambda environments.
 
See attached:
workstation.log (216 kB)
lambda.log (149 kB) - search for "InvalidClientRequest" or "REQNACK"
 
Both logs are fundamentally exercising the same code. workstation.log represents the succeeding case and lambda.log represents the failed case.
 
devin-fisher witnessed the process hang on my workstation once, but I have been unable to reproduce the hang on my workstation since.
 
The process consistently hangs in AWS Lambda. Therefore, we have a consistent way of reproducing the problem.
 
Steps to reproduce:
 
Once pull request https://github.com/evernym/sovrin-environments/pull/45 has been accepted and merged into the stable branch, follow the instructions located in https://github.com/evernym/sovrin-environments/blob/stable/vagrant/sandbox/DevelopmentEnvironment/AWS/README.md and invoke the Lambda using the following instructions:

https://gitlab.corp.evernym.com/cs/sovrin-self-serve/blob/master/lambda/nym/README.md