AWS is investigating S3 issues, affecting Quora, Slack, Trello (updated)

Cloud infrastructure provider Amazon Web Services (AWS) today confirmed that it’s looking into issues with its widely used S3 storage service in the major us-east-1 region of data centers in Northern Virginia. Other services are affected as well.

“We are investigating increased error rates for Amazon S3 requests in the US-EAST-1 Region,” AWS said at the top of its status page.

The issues appear to be affecting Adobe’s services, Amazon’s Twitch, Atlassian’s Bitbucket and HipChat, Autodesk Live and Cloud Rendering, Buffer, Business Insider, Carto, Chef, Citrix, Clarifai, Codecademy, Coindesk, Convo, Coursera, Cracked, Docker, Elastic, Expedia, Expensify, FanDuel, FiftyThree, Flipboard, Flippa, Giphy, GitHub, GitLab, Google-owned Fabric, Greenhouse, Heroku, Home Chef, iFixit, IFTTT, Imgur, Ionic, isitdownrightnow.com, Jamf, JSTOR, Kickstarter, Lonely Planet, Mailchimp, Mapbox, Medium, Microsoft’s HockeyApp, the MIT Technology Review, MuckRock, New Relic, News Corp, OrderAhead, PagerDuty, Pantheon, Quora, Razer, Signal, Slack, Sprout Social, Square, StatusPage (which Atlassian recently acquired), Talkdesk, Travis CI, Trello, Twilio, Unbounce, the U.S. Securities and Exchange Commission (SEC), The Verge, Vermont Public Radio, VSCO, Wix, Xero, and Zendesk, among other things. Airbnb, Down Detector, Freshdesk, Pinterest, SendGrid, Snapchat’s Bitmoji, and Time Inc. are currently working slowly.

Apple is acknowledging issues with its App Stores, Apple Music, FaceTime, iCloud services, iTunes, Photos, and other services on its system status page, but it’s not clear they’re attributable to today’s S3 difficulties.

Parts of Amazon itself also seems to be facing technical problems at the moment. Ironically, it’s restricting AWS’ ability to show errors.

The dashboard not changing color is related to S3 issue. See the banner at the top of the dashboard for updates.

— Amazon Web Services (@awscloud) February 28, 2017

AWS outages do happen from time to time. In 2015 an outage lasted five hours. And AWS plays an increasingly prominent role in the finances of Amazon; in the fourth quarter it yielded $926 million in operating income and $3.53 billion in revenue for its parent company.

Update at 10:30 a.m. Pacific: AWS has provided slightly more information about the S3 outage. “We’ve identified the issue as high error rates with S3 in US-EAST-1, which is also impacting applications and services dependent on S3. We are actively working on remediating the issue,” AWS said on its status page.

Update at 10:51 a.m. Pacific: AWS has another S3 status update. “We’re continuing to work to remediate the availability issues for Amazon S3 in US-EAST-1. AWS services and customer applications depending on S3 will continue to experience high error rates as we are actively working to remediate the errors in Amazon S3,” AWS said on the status page.

Update at 11:40 a.m. Pacific: A bit of good news from Amazon. “We have now repaired the ability to update the service health dashboard. The service updates are below. We continue to experience high error rates with S3 in US-EAST-1, which is impacting various AWS services. We are working hard at repairing S3, believe we understand root cause, and are working on implementing what we believe will remediate the issue,” AWS said on the status page.

Update at 11:52 a.m. Pacific: Among the services based out of Northern Virginia that are affected today, according to the status page, are Athena, CloudWatch, EC2, Elastic File System, Elastic Load Balancing (ELB), Kinesis Analytics, Redshift, Relational Database Service (RDS), Simple Email Service (SES0, Simple Workflow Service, WorkDocs, WorkMail, CodeBuild, CodeCommit, CodeDeploy, Elastic Beanstalk (EBS), Key Management Service (KMS), Lambda, OpsWorks, Storage Gateway, and WAF (web application firewall). Yikes, that’s a lot.

Update at 11:59 a.m. Pacific: AppStream, CloudWatch, Elastic MapReduce (EMR), Kinesis Firehose, WorkSpaces, CloudFormation, CodePipeline are also dealing with issues now, according to the status page.

Update at 12:06 p.m. Pacific: Some more services are down now. We’ve got API Gateway, CloudSearch, Cognito, the EC2 Container Registry, ElastiCache, the Elasticsearch Service, Glacier cold storage, Lightsail, Mobile Analytics, Pinpoint, Certificate Manager, CloudTrail, Config, Data Pipeline, Mobile Hub, and QuickSight. Wow.

Update at 12:15 p.m. Pacific: Added information on issues affecting Apple.

Update at 12:51 p.m. Pacific: On its status page, Trello just said that “S3 services appear to be slowly coming back up now.”

Update at 12:52 p.m. Pacific: And now we hear from AWS! “We are seeing recovery for S3 object retrievals, listing and deletions. We continue to work on recovery for adding new objects to S3 and expect to start seeing improved error rates within the hour,” AWS said on its status page.

Update at 1:19 p.m. Pacific: Things are looking better now. “S3 object retrieval, listing and deletion are fully recovered now. We are still working to recover normal operations for adding new objects to S3,” AWS said.

Update at 2:35 p.m. Pacific: The S3 problems have been resolved! “As of 1:49 PM PST, we are fully recovered for operations for adding new objects in S3, which was our last operation showing a high error rate. The Amazon S3 service is operating normally,” Amazon said in a 2:08 p.m. update. But several other AWS services are still having problems.

Update at 6:38 p.m. Pacific: Almost all the services affected in today’s outage are back up and running. CloudTrail, Config, and Lambda are still not fixed.

Update at 10:16 p.m. Pacific: The status now indicates that all of today’s issues have been resolved. Back to work, everyone, move along.