Summary

On September 4th, Act-On customers hosted in our European region experienced disruptions to several services, including outgoing webhooks, CRM sync, Custom Objects, dashboards and reporting, beginning at approximately 07:00 UTC (12:00 AM PDT). The issue was caused by an outage at a third-party cloud data vendor serving the Frankfurt region, and was outside of Act-On's direct control. The vendor implemented a fix, and service was confirmed restored by approximately 08:30 UTC (1:30 AM PDT).

Root cause

The incident was caused by a configuration change on our vendor's infrastructure that affected components responsible for authorizing incoming network traffic. This change caused the affected components to incorrectly restrict traffic, which in turn prevented several Act-On services from reaching our vendor's European infrastructure. The vendor resolved the issue by rolling back the configuration change.

What we are doing about it

• We are reviewing our dependency on this vendor's infrastructure to assess options for improving resilience during future vendor-side outages.
• We are evaluating whether additional monitoring or failover processes could reduce the customer-facing impact of similar third-party outages going forward.
• We will continue to work with our vendor to understand any preventative steps being taken on their end.