Downed Canvas and other Penn web services caused disruption throughout the day

Canvas and some Penn web services, as well as a number of websites across the internet, experienced outages throughout the day on Tuesday. 

These network issues stemmed from Amazon Web Services, the nation's largest cloud computing company, which was experiencing "increased error rates" for their servers. Several Penn services that utilize AWS, including but not limited to Canvas, Zoon and Panopto, were affected by this outage that turned out to be a large-scale networking issue.

Canvas announced that it was experiencing an outage that they determined was caused by AWS at 12:50 p.m. on Tuesday — many users were not been able to access the site. An update from Canvas at 2:29 p.m. announced that as Amazon was working to restore availability in their servers, Canvas’ own DevOps team was trying to expedite the process to restore access to Canvas.

“People were coming in, clutching their laptops in distress, running over to consult with me in hushed tones about their concern over not being able to submit on Canvas," English professor Caroline Whitbeck said. "I was happy to see everyone being so conscientious about deadlines, but I was sorry to see it causing distress.”

At 2:35 p.m., the services dashboard on AWS site announced that they "believe [they] understand root cause, and are working on implementing what [they] believe will remediate the issue." 

An hour an a half later at 3:52 p.m., the services dashboard on the AWS site announced that they were "seeing recovery for S3 object retrievals, listing and deletions," and that they would "continue to work on recovery for adding new objects to S3 and expect to start seeing improved error rates within the hour."

Canvas announced two minutes later that they were "beginning to see positive indications of recovery and have successfully tested workflows that were previously failing. We are still awaiting full resolution, and we will provide updates as the situation continues to improve."

 At 4:15 p.m., the Canvas website said that performance and service recovery continued to progress quickly, although there were still areas of impaired functionality as the team continued to work through the issue. The latest update at 4:37 p.m. added that the biggest area of impact was uploads, including issues such as student uploads to assignments and instructor grade uploads. 

The outages were incredibly widespread, affecting websites like Slack, Quora and Trello. USA Today reported that companies including Airbnb, Netflix, Pinterest and Spotify also rely on AWS and have experienced slowdowns. The Daily Pennsylvanian website, which also relies on AWS, was also experiencing problems displaying photos, although this was resolved by late Tuesday afternoon.

The Amazon Web Services Twitter posted updates as the situation developed.

For S3, we believe we understand root cause and are working hard at repairing. Future updates across all services will be on dashboard.— Amazon Web Services (@awscloud) February 28, 2017