Resolved -
This incident has been resolved.
Oct 21, 05:12 EDT
Update -
At this time all Wyng applications and services are operating normally. Amazon Web Services continues to work on full resolution of the root cause incident, which remains open as of the last update, at 5:48 PM U.S. Eastern Time. We are continuing to monitor.
Oct 20, 18:13 EDT
Update -
Amazon Web Services continues to work to resolve the outage, and we are continuing to monitor.
We have identified a new, related issue, which is that users editing experiences cannot view or edit sections with formatted text content. This is due to the AWS outage impacting a third-party vendor whose formatted text editor component we use. While this issue is outstanding, we recommend users hold off on editing and publishing changes to experiences that use formatted text content. Formatted text will continue to display normally in any experiences that are already live.
Here is the latest update from AWS, posted at 3:15 PM U.S. Eastern time: "[12:15 PM PDT] We continue to observe recovery across all AWS services, and instance launches are succeeding across multiple Availability Zones in the US-EAST-1 Regions. For Lambda, customers may face intermittent function errors for functions making network requests to other services or systems as we work to address residual network connectivity issues. To recover Lambda’s invocation errors, we slowed down the rate of SQS polling via Lambda Event Source Mappings. We are now increasing the rate of SQS polling as we experience more successful invocations and reduced function errors. We will provide another update by 1:00 PM PDT."
Oct 20, 15:48 EDT
Monitoring -
Amazon Web Services continues to work on full resolution, but at this time, all Wyng services are operational or recovering. Integrations and web hooks are operating normally, and all queued transactions have been processed. Weekly reports have been delivered. We continue to monitor.
Here is the latest update from AWS, posted at 12:13 PM U.S. Eastern time: "[09:13 AM PDT] We have taken additional mitigation steps to aid the recovery of the underlying internal subsystem responsible for monitoring the health of our network load balancers and are now seeing connectivity and API recovery for AWS services. We have also identified and are applying next steps to mitigate throttling of new EC2 instance launches. We will provide an update by 10:00 AM PDT."
Oct 20, 12:56 EDT
Update -
Amazon Web Services continues to work on resolving the outage, and we continue to track updates and monitor for recovery. Here is the latest update from AWS, posted at 11:04 AM U.S. Eastern time: "[08:04 AM PDT] We continue to investigate the root cause for the network connectivity issues that are impacting AWS services such as DynamoDB, SQS, and Amazon Connect in the US-EAST-1 Region. We have identified that the issue originated from within the EC2 internal network. We continue to investigate and identify mitigations.".
Oct 20, 11:39 EDT
Update -
We have updated the status of some components. We will post another update in 60 minutes.
Oct 20, 09:55 EDT
Identified -
There is a disruption affecting some Wyng customers and services. The root cause is an Amazon Web Services outage. Currently the following Wyng components and features are delayed or not working:
* Experience Metrics
* Weekly performance reports sent by email
* Content Streams
* Integrations and Webhooks
* Identity Resolution by Email and SMS
* Experiences with Custom Domains
AWS indicates that impacted services are in process of being restored. AWS Service health can be found here: https://health.aws.amazon.com/health/status
Oct 20, 09:41 EDT