Summary of the October 21, 2013 Service Failure Event

We would like to share more details with our customers about yesterday’s service failure. The service disruption began at 4:47 PM EDT on October 21st when a portion of our web services layer was logically deleted. This layer is a core infrastructure component of the FolderGrid encryption grid and used to appropriately route and authorize incoming service requests. The layer was deleted by a maintenance process that was inadvertently run against a live web service instance. The ramifications of the errant command were immediately recognized and our technical team began work to repair the damage.

FolderGrid was designed from the ground up to prioritize reliability over availability and thus reacted to this infrastructure failure by forcing our service offline and ensuring that no customer data would be at risk as the infrastructure was repaired. Unfortunately the repairs required three hours and twenty five minutes during which FolderGrid was globally unavailable. At 8:02 PM EDT, the web service layer was fully repaired and service immediately restored.

Over the coming days we will be working to ensure this type of maintenance error can not recur to protect our customers from this sort of disruption in the future. This failure resulted in our longest service outage since FolderGrid became generally available over two years ago. We know how critical our services are to our customers’ businesses, and we know this disruption came at an inopportune time for some of our customers. We will do everything we can to learn from this event and use it to drive further improvement to our service.

Sincerely,
The FolderGrid Team

Share