The aftermath of the blog platform migration

Artikkeli
06/02/2010

Well that was one long, long week…

My plan following the upgrade was to post about how it went and to introduce everyone to the new features of the site. And now it’s 9 days later… oh well, you live, you learn, you try to get some sleep once in a while.

I’m going to share some details with you but in return for that I’m also going to turn off anonymous commenting to this blog. If you want to say something, don’t hide and be anonymous. Step up and show your face rather than being a troll in the dark. I think it’s only fair that if I’m honest, you’re honest too.

D-Day… that’d be deployment day

The day started out really well. We had all the bits and pieces in place and ready to do our final deployment. Our bloggers had spent time over the weekend getting their updated designs in place and making sure their content was up to date. We began the deployment around 12pm and started deploying the final bits. Shortly after that we flipped the switch to let you, our blog readers in on the new design… and that is when things didn’t go exactly to Plan A. Well, Plan A for MSDN. The TechNet blogs didn’t really run into much trouble – we deployed, and given the scope of the content and the visitors, we were able to stay up and remain online…

However, MSDN blogs was another story. Over the next 20-30 minutes we saw our SQL Server (really sweet piece of hardware) start to get backed up with queries. That in turn backed up the web servers and before we knew it we were locking people out of the site. We removed a few widgets from the site that we knew could have an impact to performance and it helped a bit… but not enough.

We kicked into investigation mode while discussing in the background how long we had before we rolled back to the old version of the blogs. Yes, that was in our list of plans should things not get resolved quickly. We care about our bloggers internally and you our blog readers.

As we looked at the SQL traces and the site setup we found a page within the site that contained a full site tag cloud. The page was in the out-of-the-box platform deployment and just happened to be sitting at a URL the previous version of the blogs had registered with the search engines. Well, bad timing and all and the search engines decided that crawling that page right as we were going live was the best idea they’d had all day. Well every tag in the site turns out to be a lot for the MSDN Blogs and the resulting search crawl turned browsing the blogs into a crawl. Once we removed that page, things settled down for us.

After that we saw one more spike that afternoon. We have a full suite of analytics installed with the blogs. And we’d been running those for weeks without any issues. However after the search engines began their crawl the analytics system wanted to process all those page views and activity and that shot the analytics into overdrive. While the blogs remained online, we certainly were suffering from some very slow page loads. We’ve since moved any batch processing of the analytics into replicated databases to avoid this issue moving forward. Lessons learned.

Well after that we started to settle down. Well sort of. When a new blogging platform comes online not only does everyone outside Microsoft want to look at it, the 15,000 or so bloggers we have internally all decided it was time to post new articles. You just can’t script a movie story line like this. Anyway, all those bloggers posting and updating the blog designs took a bit of a toll on the system as well.

Finally around 4am I went home. Exhausted but at least we had the blogs migrated and accessible…

The days that followed…

We spent the next few days working through additional performance investigations and listening to the literally hundreds of people telling us how much we sucked for moving their cheese. We heard complaints about the W3C validation. We heard complaints about the rolling blog post list no longer existing. Heck, I think I even saw a few demands for someone to be fired… eeek!

Clearly folks are passionate about reading the blogs, which is great. However contrast what was going on outside the blogs and you come to realize this is all just very small. Oil was pouring out of the Gulf of Mexico and BP couldn’t stop it – if I thought I was getting harassed, I can’t imagine what they are dealing with. A 1000 dead Americans in the war. I even contrast it to my wife… she’s a counselor and she is dealing with 16 year olds who have no real home, no family support and are pregnant. Again, the blogs… small, really small in comparison to what’s going on in the rest of the world.

So over the past week we’ve assisted our bloggers get used to the new system. We’ve also deployed some performance fixes to both the blog sites. You should find both are now performing quite well even under load. We aren’t done but at least we are stable.

And today we’ve been able to bring back the rolling blog post list. No, it’s not on the home page – but it’s linked off there so it’s easy to get to. There were clearly some people very mad about this functionality not being online – however it was one of the casualties of the early performance issues we experienced. We tweaked a few SQL indexes and now the control is back. For some of you this is the only thing you cared about on the entire blog site – but based on our metrics, you’re in the minority. Most people find the blog content through search (external and internal to the blogs) and by direct visits to a blog. Anyway, it’s back now so hopefully that’ll help you sleep at night.

What’s up with the home page?

The home page has been updated to move beyond just the rolling blog post list. As a landing page to the blogs, new visitors need to land softly. The old home page, while clearly useful for some, actually was one of the weirdest experiences for new users I’ve seen. It didn’t even look like MSDN or TechNet. The traffic to the home page told a clear story – yes, some folks landed there, but that was less than 1% of our traffic. We have many blogs that receive more traffic than that. And if we wanted to move the blogs on to 2010, then we needed to look at the data and adjust from there. Which we’ll continue to do…

So we have a few things on the home pages now…

An editorial section. Right now there is a welcome message. As time goes on we’ll be updating that to be in alignment with spotlights and features from the corresponding MSDN and TechNet sites. We’ve clearly heard the feedback that we need to remove the silo’s of information and this is one step forward to bring consistency between our applications and sites.
Recent Blog Posts. The recent blog posts is a fast way to see what’s been posted recently and expand the reach of the RSS feed for the blogs. Many, many people follow the blogs solely via RSS and that’s great. We want to encourage that. You’ll now find the link to the rolling blog post list at the bottom of the Recent Blog Post widget.
Recent Site Activity. This lets you know what’s going on in the site, with the people you’re following and the activity you’ve performed. Perhaps you commented on a blog but can’t remember which one, just look at your activity and you’ll be quick to return. Again, as a landing page for the blogs this allows people to see the site is active and not be overwhelmed by a never ending list of posts.

If you’re one of those who just want to the old blog post list back then use the new rolling blog post list page. There is nothing else on the page to distract you. Enjoy. :-)

What’s next?

So at this point we’ve got the new blog site where we expected it to be about a week ago. We’ll be updating the help pages and finally getting to talk about the new features. Our bloggers are mostly up to speed now and I’m seeing a lot of posts come out daily which is great.

So what’s next… well we are working on the next update which should be out in a couple of weeks from now. This should include:

Bug Fixes: Yep, there are bugs. We’ve been logging them and we are working on fixing them.
Performance Improvements: There is still yet more performance we can wring out of this platform. We’re analyzing load times, SQL traces, CSS and HTML to reduce what we can.
Search within a Blog: Right now if you search while viewing a blog, you’ll get search results from all blogs. That seems like a great idea if you’re just looking for an answer if you don’t care where it comes from. But we also know searching within a blog is a great feature so that’s on the schedule as well.
Default discovered RSS Feed on a Blog: Right now it’s broken. It points to a search RSS feed that doesn’t exist. This one slipped through. We’ll correct this to be the posts for a blog.
A few new widgets: Our bloggers will have a few new widgets to use including a standard code syntax highlighter along with the Microsoft Translation widget.
Validation of HTML:We have identified a few issues with the HTML being output. We’ll fix those over time as we can.

Ok, with that I’m definitely late for dinner. Hopefully this post won’t cause a riot and we can have constructive conversations about the blogs, not just shouting, flaming abuse sent my way. I want to listen and I want the blogs to be a great place for all of us. Let’s keep the talk respectful and the comments on topic and we can move forward together…

Thanks,
Sean

Comments

Anonymous
June 02, 2010
The comment has been removed
Anonymous
June 02, 2010
@Trees: use the 'BlogsOnly' URL: blogs.msdn.com/.../mainfeed.aspx
Anonymous
June 02, 2010
Hi Sean, for performance and (everybodys!) bandwidth, what about HTTP Conditional Get (by timestamp/ETag: 304) for the RSS/Atom feeds? Thanks!
Anonymous
June 02, 2010
I like your narative about how the upgrade went. Our team just upgraded ITWeb here at MS and although it went smooth, we had our own issues. I might post something like this that tells the story of how cool it was to share desktops with PMs and Developers in China during our sync ups about the project. We had a couple odd issues that held us up until 3am but that's IT. Nice work and thanks for all that. I would really like to see non-FTE's who have expertise and experience working here at MS as "a-" and "v-" folks to be able to blog on MSDN though. We have to have an FTE own the blog which means we don't control it when we leave.
Anonymous
June 03, 2010
It would be nice on the rolling blog post list page to have different sytles for visited blog entries. Seems that right now they are painted with the style as unvisited ones.
Anonymous
June 03, 2010
What's with the /b/ prefix? Is this a sharepoint 2010 platform, and the "b" is a bucket to get around the hard limits around folder items? If it's 2010, did this migration experience go towards sp1 improvements for sp2010? Any other technical design detials available?
Anonymous
June 03, 2010
The comment has been removed
Anonymous
June 03, 2010
oh, real feedback... thanks everyone :-) To answer a few questions...

The /b in the URLs: The platform we are using is the Telligent Community Server platform. The platform can do more than just blogs (wiki's, forums, media galleries etc.) and each of those gets its own namespace in the platform. We spent a good deal of time looking into how to remove, hide or obstruct the /b from the URL but in the end the cost and time was very prohibitive and should we do it, we'd be on our own with the platform. It'd be tough to pick up hotfixes and changes from Telligent. In the end the best decision for future platform development and support was to leave the /b in.
The Blog Post: prefix. As Thomas said, there is a different feed now for just the blogs. The main site feed at /rss.aspx combines users and blogs (and if the platform was also hosting wikis, forums etc. would be pumping out those into that main list.)
Perf and Etags: We'll see if we can get them on the feeds. Good idea.
Rolling Blog Post Colors: Yes, that's a great idea. Now I just need to find a color... :) On the sharing desktops, we've been doing a lot of that since launch. It really is great technology and really speeds up solving problems for people. It's just like being there I guess :)

Anonymous
June 03, 2010
One of the things you promised before the conversion was better HTML closer to the standards. It was surpsing that the HTML validation seemed to be worse (something we would have expected to have been tested). I still would like to see direct links to the comment on the frontpage of blog Both next to the article for short preview themed frontpage (like your blog) but also below the article for blogpages that show full posts on the frontpage.(like the IE blog) I would also prefer the blog reader to be in control of the number of comments per page.
Anonymous
June 04, 2010
Thank you for the reply I thought there had to be more than just Blog Post and User. Now htat I think of them Twitter? News, Promos? Maybe and explaination of what they are and how we can filter. Where did Thomas find out about BlogOnly? But now I'm confused I did some testing and found there are different URLs? blogs.msdn.com/MainFeed.aspx redirects to /rss.aspx. Has a title page that is generic "Site Home" This URL shows the prefix, Shows User: and Page: It is as if this is the "New Look (in progress)" tied to the old URL. blogs.msdn.com/.../mainfeed.aspx title page is "MSDN Blogs" Which does not show the prefix, does not show User: nor Page: and I did not need the parameter BlogsOnly. It is as if this is the "Old Look" tied to a new URL. IMHO I would have had them switched it is as if we are forced the new look and the old look can be had by using a new URL. Instead of havig the new URL point to the new site and leave the link that everyone uses alone. You moved my cheese ;) just joking. Why even have the two URLs? How about making the MainFeed.aspx redirect to /b/mainfeed.aspx instead of /rss.apsx? Ohh wait then we would be back to the old look. I guess the solution is to change links and rss feeds from /mainfeed.aspx to /b/mainfeed.asxp to get the old look and feel? But then we will not get Page: or Twitter or Forum: or any other Group that comes along? Na what I'm going to do is a have a bit of fun with this one. I'm add them both to my Favorite Bar and watch them (my cheese) move over time. Keep up the good work! I know it is these little things can seem to be a bin pain after the huge accomplishments you have made. As I was writing this a new one popped up Walkthrough: Build, Host, and Test Simple RESTful WCF Service. The funny thing is that was with the BlogsOnly. Now I'm really confused!!!! Then if I go back to /rss.aspx I get (test BlogOnly) Blog Post: Walkthrough: Build, Host, and Test Simple RESTful WCF Service A Blog Post and a Walkthrough. I'm done the more I try to explain as data is being posted I get more confused.
Anonymous
June 04, 2010
I could not put this down for more than five minutes. blogs.msdn.com/.../mainfeed.aspx Is the way to go. Except the Type=BlogOnly does not work. The concept I believe goes along the lines of what I was thinking before. There might not be any other type so why have a prefix, overkill. The majority of these will be blog posts so why prefix them all with Blog Post. This what you get when you use /b/mainfeed.aspx. It only prefixes those that are not a Blog Post. That is why the post about Build, Host, and Test Simple RESTful WCF Service is prefixed with Walkthrough because it is not a Blog per say but a Walkthrough. If this is the desired effect than the /rss.aspx or /mainfeed.aspx is prefixing everything with BlogPost regardless. That is why there is duplication of the prefix? The Build, Host, and Test Simple RESTful WCF Service was prefixed with Walkthrough but it is as if /rss.aspx is prefixing everything regardless then Walkthrough: Build, Host, and Test Simple RESTful WCF becomes Blog Post: Walkthrough: Build, Host, and Test Simple RESTful WCF Service
Anonymous
June 04, 2010
The comment has been removed
Anonymous
June 04, 2010
And as one would guess with /rss.aspx it appears as Blog Post: Automatic Reply: Radio silence for a bit and on /b/mainfeed.aspx it appears as Automatic Reply: Radio silence for a bit
Anonymous
June 07, 2010
Recent Blog Posts not functioning again...just displays: blogs.msdn.com/.../mainfeed.aspx
Anonymous
June 07, 2010
@StealthTech - I totally appreciate the feedback and the information you're collecting. We are finding new scenarios everyday! I think when you see something like "Automatic Reply:" or "Walkthrough:" that's probably the name of the post not neccessarily a group or action on the site. Though that Automatic reply one is an interesting usage case, I guess it could be someone on vacation ;-) We are going to switch the RSS feed redirect so that the original mainfeed.aspx goes to the same content in the new place. That's what you said in your comment if I'm confusing here :) That'll be coming in an update in the next few weeks. BTW, it's Type=BlogsOnly (not sure if it was a typo to have BlogOnly)... @Kizzer - thanks for reporting it. We are fixing a time out issue on the control that renders that RSS feed list. Hoping to have that fixed in the upcoming weeks as well.
Anonymous
June 08, 2010
Your welcome Sean, There have been some intermittent issues with displaying the RSS list in IE 8 due to some posting having errors. Therefore my list gets grayed out. It happens sometimes, but seems to be happening more often than before. Doing some more testing today related to my typo. Here are my use cases (time/day dependant done at 11:22 a.m -8 time zone) If I go to blogs.msdn.com/.../mainfeed.aspx I get 62 out of 62 Regardless of what Parameter I choose makes no difference. That is why I think the typo did not matter. If there is a Question Mark at the end of the URL the query is thrown off: blogs.msdn.com/.../mainfeed.aspx I get 15 posts which is no different if I use a typo such as "Type=BlogOnly" or use the real deal "Type=BlogsOnly " I get 15 posts. Just as much as "Type=AllBlogs" clicked on from the MSDN Blog home page. I get 15 posts. You raise a question with your answer about the Walkthrough prefix. How are end users going to know what prefixes relate to a "group or action" and what is a user prefix? How was I to know that "Walkthrough:" was entered by the blog poster and not the RSS feed trying to put the post into a group called "Walkthrough:"? I agree that htis was a user entry because if you view the Blog you will see in the Title Walkthrough. But because the formating of the text with a collen was the same as how prefixes are done it was not obvious. You are spot on with the Redirect. I will be looking for that to come. I know how I was explaining was confusing, but that was part of the point. So confusing it is hard to explain ;). And yes the ideal solution is to have all links go to the same place and not have too many different looks. At the moment how many URLs to the RSS feed? I count three /rss.aspx, /MainFeed.aspx, and /b/MainFeed.aspx. I'll keep posting if I find anything.
Anonymous
June 21, 2010
Hope all is going well. I have yet to see any changes and was just wondering if it is me or something with the RSS Feeds. I mentioned this in previous posts. The list in IE 8 is grayed out more often than not. When I click the link to open all items in IE I get the message below. This only goes away when there are no items with an error. It just seems that 9 out 10 items on the list are in error and therefore there is a low probability that the list will not be grayed out. Almost every single day and throughout the day it is grayed out. "This feed contains errors. Internet Explorer will try updating this feed again later."
Anonymous
June 21, 2010
Hope all is going well. I have yet to see any changes and was just wondering if it is me or something with the RSS Feeds. I mentioned this in previous posts. The list in IE 8 is grayed out more often than not. When I click the link to open all items in IE I get the message below. This only goes away when there are no items with an error. It just seems that 9 out 10 items on the list are in error and therefore there is a low probability that the list will not be grayed out. Almost every single day and throughout the day it is grayed out. "This feed contains errors. Internet Explorer will try updating this feed again later."
Anonymous
September 07, 2010
The comment has been removed
Anonymous
September 07, 2010
@DWalker59 - Totally understand what you are saying - it's definitely something we have been talking about here in the MSDN/TechNet team (which is all the same team). We absolutely realize that folks are more than just developers or IT Pro's and that we need to evolve our approach to the current industry. Stay tuned, we'll keep working on it :)

Jaa