tag:blogger.com,1999:blog-30604802024-02-08T08:21:28.051-08:00Blogger StatusWhat's up with Blogger development and serversUnknownnoreply@blogger.comBlogger736125tag:blogger.com,1999:blog-3060480.post-59193172703462858212013-06-04T18:20:00.001-07:002013-06-04T18:20:31.462-07:00<div dir="ltr" style="text-align: left;" trbidi="on">
<span style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.727272033691406px;">This blog has been retired. The latest information on Blogger service availability will now be displayed on the </span><a href="http://www.google.com/appsstatus" style="background-color: white; color: #1155cc; font-family: arial, sans-serif; font-size: 12.727272033691406px;" target="_blank">Apps Status Dashboard</a><span style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.727272033691406px;">.</span></div>
Unknownnoreply@blogger.comtag:blogger.com,1999:blog-3060480.post-71223456885398779602011-08-18T07:58:00.000-07:002011-08-18T09:45:41.182-07:00<span class="Apple-style-span" style="background-color: white; font-family: arial, sans-serif; font-size: 13px;">We have received reports from users not being able to create or edit posts. We are working on fixing this. In the meantime, you can change your language settings to English as a workaround. </span><span class="Apple-style-span" style="background-color: white; font-family: arial, sans-serif; font-size: 13px;">Thanks for your patience.</span><br />
<span class="Apple-style-span" style="background-color: white; font-family: arial, sans-serif; font-size: 13px;"><br />
</span><br />
<span class="Apple-style-span" style="background-color: white; font-family: arial, sans-serif; font-size: 13px;"><b>Update, 9:45AM PST: </b>This has been fixed, thanks for your patience.</span>Unknownnoreply@blogger.comtag:blogger.com,1999:blog-3060480.post-9663013758068471222011-08-09T12:54:00.000-07:002011-08-10T18:13:38.461-07:00Blogger will go into read-only mode <b>Wednesday (8/10) 5:00PM PST</b> for about an hour as we work to expand capacity for our users. Thanks in advance.<br />
<br />
<b>Update, 6:10PM:</b> The maintenance is complete, thanks for your patience.Unknownnoreply@blogger.comtag:blogger.com,1999:blog-3060480.post-18616475668595562862011-07-10T12:26:00.000-07:002011-07-10T18:00:08.926-07:00We're investigating reports of 503 errors this morning, and hope to have the issue resolved shortly. We'll update this thread once we have more information to share.<br />
<br />
<b>Update, 2:00PST</b>: This has been resolved, thanks for your patience.Unknownnoreply@blogger.comtag:blogger.com,1999:blog-3060480.post-8787865909982849302011-06-03T11:47:00.000-07:002011-06-06T19:02:23.522-07:00Image Uploads will be temporarily disabled for about two hours while we perform routine maintenance on <b>Monday, June 6th at 5:00PM PDT</b>.<br />
<br />
However, you will still be able to add photos to your blog via the <i>From this blog </i>and<i> From PWA </i>tab of the uploader. We'll update this post as soon as the maintenance is complete.<br />
<br />
<i>(This has been moved forward one day from the originally-scheduled date of June 7th)</i><br />
<i><br />
</i><br />
<i><b>Update, 7:00PM: </b>The maintenance is now complete. Thanks for your patience.</i>Unknownnoreply@blogger.comtag:blogger.com,1999:blog-3060480.post-50682434194763792472011-05-31T18:25:00.000-07:002011-05-31T20:26:24.308-07:00<div dir="ltr" trbidi="on"><span class="Apple-style-span" style="font-family: Arial; font-size: 15px; white-space: pre-wrap;"><b>Blogger Incident Report</b></span><br />
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">By Eddie Kessler, Tech Lead/Manager, Blogger</div><div></div><br />
<i>(This is a follow-up to the <a href="http://buzz.blogger.com/2011/05/blogger-is-back.html">original post</a> we made on Blogger Buzz about this incident.)</i><br />
<br />
On Wednesday, May 11 at 10:00pm PDT, Blogger entered a one-hour scheduled maintenance period to make improvements to increase service reliability. Unfortunately, errors made during this period had the opposite effect. This report describes what we were doing, what went wrong, how we fixed it, and what we’re doing to help prevent these types of problems from happening again.<br />
<br />
We sincerely apologize for the impact of this incident on Blogger’s authors and readers.<br />
<br />
<u>What we were doing</u><br />
<br />
Blogger maintains replicas (copies) of blogs in multiple locations so if one copy becomes unavailable — maybe due to a cut network cable or loss of power — blogs continue to remain accessible. During this maintenance period we attempted to add several more replicas to both increase redundancy (which helps make the service more robust) and capacity (so we can serve more blogs to more people).<br />
<br />
The procedure required a read-only period (no new posts, comments, or blogs) while we added the new replicas. The procedure appeared to go smoothly, and just after 10:30pm PDT we allowed new blogs, posts, and comments. We then noticed a higher than usual rate of errors being reported and quickly realized that the new replicas were missing data.<br />
<br />
<u>What went wrong</u><br />
<br />
Next, we reverted the service upgrades and removed the newly introduced replicas. During this process we discovered that running Blogger with the bad replicas had caused some user data to become inaccessible. This manifested itself in a number of ways including some blogs appeared to have been replaced, others wouldn’t display, and some users couldn’t access their dashboard.<br />
<br />
<u>Fixing the problem</u><br />
<br />
After several hours of examining different strategies to address the data problems we decided to restore Blogger data from our backup systems. First we had to restore the data from backups to our serving infrastructure and then we had to recover all the posts, pages, and comments that had been made since the backup was taken.<br />
<br />
While we restored the service from backup, Blogger remained in read-only mode for just over 10 hours, after which the majority of blogs returned to normal. The backup, however, had some inconsistencies that affected a very small percentage of blogs. Also, the process used to migrate posts immediately after restoration had some unforeseen side-effects that we had to address. <br />
<br />
Once we had all the blogs restored, we put our energy toward removing inconsistencies and gradually restoring posts, pages and comments. This required detailed work, and we focused on being meticulous to make sure that we didn't create any additional issues.<br />
<br />
<u>Lessons learned and preventative actions</u><br />
<br />
We’ve learned a lot in a short period of time about how to manage failures in our maintenance and recovery processes. We’ve identified multiple areas to fix and improve — including better tools for repairing inconsistencies in our data store, defensive elements in our software to guard against corruption, better backup and restore procedures, and some procedural changes for maintenance that would have prevented the initial issue. We also outlined how we can improve our communication with Blogger users should something like this occur again, which would include more consistent updates on the user forum.<br />
<br />
During this time, we received messages of support from some users for which we are very grateful. Blogger users have spent countless hours creating blogs that are amazing, creative, important, and personal. Those hours have weighed heavily on us, and for all the anxiety and frustration that this incident caused some Blogger users, we sincerely apologize.<br />
<br />
We are committed to swiftly fixing the problems and using our lessons learned to try to make sure that this type of issue doesn't happen again.</div>Unknownnoreply@blogger.comtag:blogger.com,1999:blog-3060480.post-72447469381389904022011-05-13T10:32:00.000-07:002011-05-13T10:35:33.693-07:00Blogger is back now. We're still working on restoring some of the data. For more details, see this post: http://buzz.blogger.com/2011/05/blogger-is-back.htmlUnknownnoreply@blogger.comtag:blogger.com,1999:blog-3060480.post-19143453075722400492011-05-13T06:07:00.000-07:002011-05-27T08:19:56.857-07:00We’ve started restoring the posts that were temporarily removed and expect Blogger to be back to normal soon.Unknownnoreply@blogger.comtag:blogger.com,1999:blog-3060480.post-23909457931612199092011-05-13T04:25:00.001-07:002011-05-13T04:25:22.347-07:00<div dir="ltr" style="text-align: left;" trbidi="on">To get Blogger back to normal, all posts since 7:37am PDT on Weds, 5/11 have been temporarily removed. We expect everything to be back to normal soon. Sorry for the delay.</div>Unknownnoreply@blogger.comtag:blogger.com,1999:blog-3060480.post-67752357861822076702011-05-12T12:34:00.000-07:002011-05-13T13:20:19.067-07:00<div dir="ltr" style="text-align: left;" trbidi="on">Blogger will be in read-only mode while we resolve some maintenance issues. Sorry for the inconvenience.</div>Unknownnoreply@blogger.comtag:blogger.com,1999:blog-3060480.post-2232272894834580492011-05-09T21:17:00.000-07:002011-05-09T21:17:19.307-07:00Blogger will go into read-only mode <b>Wednesday (5/11) 10:00PM PST</b> for about an hour for maintenance.Unknownnoreply@blogger.comtag:blogger.com,1999:blog-3060480.post-36467559663834998092011-04-12T01:25:00.000-07:002011-04-12T01:25:12.982-07:00We've identified a problem rendering some blogs containing the popular posts widget. Our engineers are fixing the issue and the it should be resolved shortly.<br />
<br />
Thanks for your patience in the mean time.Unknownnoreply@blogger.comtag:blogger.com,1999:blog-3060480.post-58386948036125292032011-03-17T10:43:00.000-07:002011-03-17T10:43:10.719-07:00<div dir="ltr" style="text-align: left;" trbidi="on"><br />
<b>Postmortem for Blogger Service Interruption on March 16</b><br />
<br />
By Jinwoo Lee, Blogger Software Engineer<br />
<br />
As <a href="http://buzz.blogger.com/2009/11/improving-our-communication-of-blogger.html">we committed to you</a>, when serious service interruptions occur we will provide you details about what went wrong and what we’re doing to help prevent similar problems in the future.<br />
<br />
From about 9:55 AM PDT until about 10:20 AM on Wednesday, March 16, all Blogger users were unable to access the Blogger dashboard page at <a href="http://www.blogger.com/">www.blogger.com</a> and <a href="http://draft.blogger.com/">draft.blogger.com</a>. No data was lost and blogspot was not affected at all, so your blog’s readers weren’t impacted.<br />
<br />
We made brief status updates on our <a href="http://twitter.com/Blogger">Twitter account</a> — <a href="http://twitter.com/Blogger/status/48071011328987136">this</a> and <a href="http://twitter.com/Blogger/status/48072154436218880">this</a>. Meanwhile, we kept a close eye on the reports coming through our <a href="http://www.google.com/support/forum/p/blogger">help forum</a> and <a href="http://twitter.com/search/%40blogger">Twitter stream</a>.<br />
<br />
The problem was caused by human error. We wanted to show a small announcement on the Blogger dashboard and our code for it had a bug, which caused the problem. Once we figured this out, we quickly reverted the change.<br />
<br />
We are truly sorry for any inconveniences this might have caused you. In the meantime, we will try harder to make Blogger a more reliable service.<br />
</div>Jinwoo Leehttp://www.blogger.com/profile/10856136370896436149noreply@blogger.comtag:blogger.com,1999:blog-3060480.post-35137450876938222582010-10-18T11:21:00.000-07:002010-10-20T18:31:06.932-07:00Image Uploads will be disabled for about two hours while we perform routine maintenance on <b>Wednesday, October 20th </b>at <b>5:00PM PDT. </b><br />
<br />
We'll update this post as soon as the maintenance is complete.<br />
<br />
<b>Update, 6:30PM: </b>This is now complete and uploads are back online. Thanks for your patience.Unknownnoreply@blogger.comtag:blogger.com,1999:blog-3060480.post-20395315788418816472010-07-06T12:00:00.000-07:002010-07-07T17:48:27.257-07:00We're currently working through multiple issues which are affecting the reliability of our comment system. We'll update this post as soon as we have more information to share.<br />
<div><br />
</div><div>Thanks for your patience in the meantime. <br />
<br />
Update: This has been resolved. </div>Unknownnoreply@blogger.comtag:blogger.com,1999:blog-3060480.post-27265775273478218802010-06-08T23:30:00.000-07:002010-06-08T23:30:03.546-07:00<b>Post Mortem for Blogger Service Interruption June 7</b><br />
<br />
By Eddie Kessler, Blogger Engineering Manager<br />
<br />
As <a href="http://buzz.blogger.com/2009/11/improving-our-communication-of-blogger.html">we recently committed to you</a>, when serious service interruptions occur, we will provide you details about what went wrong and what we're doing to help prevent similar problems in the future.<br />
<br />
From about 12:30 AM PST on Monday, June 7 until about 12:30 PM, a significant subset of Blogger users, mostly in the US Midwest, were unable to post to their blogs. No data was lost and blogspot was largely unaffected, so your blog’s readers weren’t impacted.<br />
<br />
We posted the issue on the <a href="http://blogger-status.blogspot.com/2010/06/were-aware-of-isolated-access-issues-in.html">Blogger Status blog</a>, and we made a brief status update on our <a href="http://twitter.com/Blogger">Twitter account</a> as well. Meanwhile, we kept a close eye on the reports coming through our <a href="http://www.google.com/support/forum/p/blogger/thread?tid=70655d02914de49f&hl=en">help forum</a> and <a href="http://twitter.com/#search?q=%40blogger">Twitter stream</a>.<br />
<br />
The problem was caused by human error. One of our datacenters inadvertently had the wrong server configuration when we performed our regular service update. Once we figured this out, we corrected the problem quickly.<br />
<br />
We are in the process of adding additional monitoring that will prevent this kind of outage from recurring.<br />
<br />
We are truly sorry for any inconveniences this might have caused you. In the meantime, we will try harder to make Blogger a more reliable service.Unknownnoreply@blogger.comtag:blogger.com,1999:blog-3060480.post-35669832099167945312010-06-07T09:49:00.000-07:002010-06-07T12:42:25.765-07:00We're aware of isolated access issues in certain regions within the US. We're investigating this now and will follow-up as soon as we have more information to share<br /><br />Thanks for your patience in the meantime.<br /><br /><span style="font-weight: bold;">Update, 12:30 PST: </span>This is now fixed.Unknownnoreply@blogger.comtag:blogger.com,1999:blog-3060480.post-50815101137287977782010-03-29T09:49:00.000-07:002010-03-29T15:30:02.285-07:00Our image backend is experiencing problems which may affect the display and/or uploading of photos. We are working to resolve this and will update this post when we have more information.<br />
<br />
More info can be found here in the <a href="http://www.google.com/support/forum/p/Picasa/thread?tid=2f5092e53d4653e5&hl=en">Picasa Help Forum</a>.<br />
<br />
<b>Update, 3/29 2:00PST: </b>This is resolved. Thanks for your patience.Unknownnoreply@blogger.comtag:blogger.com,1999:blog-3060480.post-1167341620569113452010-01-25T13:42:00.000-08:002010-01-25T13:44:54.139-08:00<span dir="ltr" id=":2lk">The Following feature will be read-only for planned maintenance for about two hours starting <span style="font-weight: bold;">Monday (1/25) at 2:00PM PDT</span>. Any new Followers or other Friend Connect changes may not save during this time.</span>Unknownnoreply@blogger.comtag:blogger.com,1999:blog-3060480.post-43212883636807632182010-01-19T22:50:00.001-08:002010-01-22T10:21:01.400-08:00Users are experiencing problems playing videos in Blogger blogs. We are investigating and will send an update when fixed.<div><br /></div><div><b><i>Update 1/22: <span class="Apple-style-span" style="font-weight: normal;"><span class="Apple-style-span" style="font-style: normal;">This is now fixed. Thanks for your patience.</span></span></i></b></div>Unknownnoreply@blogger.comtag:blogger.com,1999:blog-3060480.post-46281896953886992942009-11-23T11:21:00.000-08:002009-11-23T17:00:22.099-08:00<span style="color: #333333; font-family: 'trebuchet ms';">Image uploads will be unavailable Monday (11/23) at 4:00PM PDT for about 30 minutes for maintenance.</span><br />
<span style="color: #333333; font-family: 'trebuchet ms';"><br />
</span><br />
<span style="color: #333333; font-family: 'trebuchet ms';">Update: This is now complete. Thanks for your patience.</span>Unknownnoreply@blogger.comtag:blogger.com,1999:blog-3060480.post-58903986383072907402009-11-04T20:03:00.000-08:002009-11-04T20:18:25.714-08:00Postmortem on recent issues affecting users in Europe<br />
<br />
<b>What happened</b> <br />
Beginning at approximately 2:30am (PDT) Saturday, October 31, a DNS configuration issue in a new datacenter in Europe caused requests to blogger.com and *.blogspot.com to be sent to an invalid IP address. Affected users throughout Europe were unable to connect to either Blogger or to blogs hosted by Blogger at blogspot.com for approximately 10 hours. <br />
<br />
<b>What we're doing about it</b><br />
1. Blogger's systems engineers will proactively monitor and verify DNS settings during major system changes.<br />
2. We have improved our automated monitoring so that on-call engineers will be paged earlier for DNS-related failures.<br />
3. Checks in our DNS management system have been made to prevent the change that caused this to occur.<br />
<br />
Eddie Kessler, Blogger Engineering ManagerEddiehttp://www.blogger.com/profile/09007008117507956805noreply@blogger.comtag:blogger.com,1999:blog-3060480.post-91054421016827934872009-10-31T12:42:00.000-07:002009-11-04T18:57:39.672-08:00RESOLVED: The issue affecting Blogger users in Europe has been resolved. There was a configuration problem in a new datacenter. We will provide a postmortem on Monday. Thanks to all who assisted us analyzing the problem.Eddiehttp://www.blogger.com/profile/09007008117507956805noreply@blogger.comtag:blogger.com,1999:blog-3060480.post-6418220058533388542009-10-31T10:50:00.000-07:002009-10-31T10:50:16.482-07:00We are receiving reports that some users cannot access Blogger in Europe. We are investigating.Unknownnoreply@blogger.comtag:blogger.com,1999:blog-3060480.post-7949970190982146552009-10-23T01:07:00.000-07:002009-10-23T01:07:13.680-07:00Blogger was unavailable for most users for over an hour. Service has now been restored.Eddiehttp://www.blogger.com/profile/09007008117507956805noreply@blogger.com