An issue with event processing streams caused delays in integration processing affected a subset of Wyng clients. Initial behavior was intermittent, with delays occurring sporadically on July 1, that progressed to a backlog of up to 9 hours of transactions for affected clients starting at about 4:00 AM EDT on July 2.
Wyng ops staff took initial action on the issue at 8:30 AM EDT on July 2. By 12:00 EDT the backlog reduced to 2 hours, by 4:20 PM EDT the backlog was 1 hour, and the issue was resolved for all affected clients at 6:38 PM EDT on July 2.
The root cause was interdependence between a set of clients whose transactions were being processed in a shared stream. Significant delays in downstream processing by one client caused a backlog across all clients sharing the stream.
To avoid recurrence of this failure mode, Wyng is prioritizing a infrastructure update to move from shared to dedicated streams for transaction event processing.