Tuesday, June 04, 2013

This blog has been retired. The latest information on Blogger service availability will now be displayed on the Apps Status Dashboard.

Thursday, August 18, 2011

We have received reports from users not being able to create or edit posts. We are working on fixing this. In the meantime, you can change your language settings to English as a workaround. Thanks for your patience.

Update, 9:45AM PST: This has been fixed, thanks for your patience.

Tuesday, August 09, 2011

Blogger will go into read-only mode Wednesday (8/10) 5:00PM PST for about an hour as we work to expand capacity for our users. Thanks in advance.

Update, 6:10PM: The maintenance is complete, thanks for your patience.

Sunday, July 10, 2011

We're investigating reports of 503 errors this morning, and hope to have the issue resolved shortly. We'll update this thread once we have more information to share.

Update, 2:00PST: This has been resolved, thanks for your patience.

Friday, June 03, 2011

Image Uploads will be temporarily disabled for about two hours while we perform routine maintenance on Monday, June 6th at 5:00PM PDT.

However, you will still be able to add photos to your blog via the From this blog and From PWA tab of the uploader. We'll update this post as soon as the maintenance is complete.

(This has been moved forward one day from the originally-scheduled date of June 7th)

Update, 7:00PM: The maintenance is now complete. Thanks for your patience.

Tuesday, May 31, 2011

Blogger Incident Report
By Eddie Kessler, Tech Lead/Manager, Blogger

(This is a follow-up to the original post we made on Blogger Buzz about this incident.)

On Wednesday, May 11 at 10:00pm PDT, Blogger entered a one-hour scheduled maintenance period to make improvements to increase service reliability. Unfortunately, errors made during this period had the opposite effect. This report describes what we were doing, what went wrong, how we fixed it, and what we’re doing to help prevent these types of problems from happening again.

We sincerely apologize for the impact of this incident on Blogger’s authors and readers.

What we were doing

Blogger maintains replicas (copies) of blogs in multiple locations so if one copy becomes unavailable — maybe due to a cut network cable or loss of power — blogs continue to remain accessible. During this maintenance period we attempted to add several more replicas to both increase redundancy (which helps make the service more robust) and capacity (so we can serve more blogs to more people).

The procedure required a read-only period (no new posts, comments, or blogs) while we added the new replicas. The procedure appeared to go smoothly, and just after 10:30pm PDT we allowed new blogs, posts, and comments.  We then noticed a higher than usual rate of errors being reported and quickly realized that the new replicas were missing data.

What went wrong

Next, we reverted the service upgrades and removed the newly introduced replicas. During this process we discovered that running Blogger with the bad replicas had caused some user data to become inaccessible. This manifested itself in a number of ways including some blogs appeared to have been replaced, others wouldn’t display, and some users couldn’t access their dashboard.

Fixing the problem

After several hours of examining different strategies to address the data problems we decided to restore Blogger data from our backup systems. First we had to restore the data from backups to our serving infrastructure and then we had to recover all the posts, pages, and comments that had been made since the backup was taken.

While we restored the service from backup, Blogger remained in read-only mode for just over 10 hours, after which the majority of blogs returned to normal. The backup, however, had some inconsistencies that affected a very small percentage of blogs.  Also, the process used to migrate posts immediately after restoration had some unforeseen side-effects that we had to address.

Once we had all the blogs restored, we put our energy toward removing inconsistencies and gradually restoring posts, pages and comments. This required detailed work, and we focused on being meticulous to make sure that we didn't create any additional issues.

Lessons learned and preventative actions

We’ve learned a lot in a short period of time about how to manage failures in our maintenance and recovery processes. We’ve identified multiple areas to fix and improve — including better tools for repairing inconsistencies in our data store, defensive elements in our software to guard against corruption, better backup and restore procedures, and some procedural changes for maintenance that would have prevented the initial issue. We also outlined how we can improve our communication with Blogger users should something like this occur again, which would include more consistent updates on the user forum.

During this time, we received messages of support from some users for which we are very grateful. Blogger users have spent countless hours creating blogs that are amazing, creative, important, and personal. Those hours have weighed heavily on us, and for all the anxiety and frustration that this incident caused some Blogger users, we sincerely apologize.

We are committed to swiftly fixing the problems and using our lessons learned to try to make sure that this type of issue doesn't happen again.

Friday, May 13, 2011

Blogger is back now. We're still working on restoring some of the data. For more details, see this post: http://buzz.blogger.com/2011/05/blogger-is-back.html
We’ve started restoring the posts that were temporarily removed and expect Blogger to be back to normal soon.
To get Blogger back to normal, all posts since 7:37am PDT on Weds, 5/11 have been temporarily removed. We expect everything to be back to normal soon. Sorry for the delay.

Thursday, May 12, 2011

Blogger will be in read-only mode while we resolve some maintenance issues. Sorry for the inconvenience.

Monday, May 09, 2011

Blogger will go into read-only mode Wednesday (5/11) 10:00PM PST for about an hour for maintenance.

Tuesday, April 12, 2011

We've identified a problem rendering some blogs containing the popular posts widget.  Our engineers are fixing the issue and the it should be resolved shortly.

Thanks for your patience in the mean time.