Description
This is a follow-up to INDY-366. In that ticket, we changed the default FIFO queue in ZMQ to be unlimited. This makes certain problems less likely, but it also masks problems for a lot longer. I would be more comfortable that we were truly robust if:
- The high water mark were finite (e.g., 100,000).
- We had a way to detect that we are nearing the high water mark, and that we would report a graceful error that makes this condition easy to diagnose.
- We implemented logic in STP to discard old messages rather than new ones. (Desirable but not necessarily required.)
- If we exceed the high water mark, we exit the daemon process gracefully rather than crashing due to an out-of-memory condition.
This ticket is to track the work in items 1-4. Possibly 3 could be omitted.