Longest remote home folder test … 600 miles…
Sorry it’s been so long since my last post, but I’ve been quite busy fixing bugs which are either old, new, borrowed and quite far away. In fact some are so far away it takes driving all the way to Burbank, CA to verify the fix.
I’ve spent a good number of weeks working on fixing a variety of Remote Home Folder bugs. If you are not sure what I am talking about you can read about it here: Apple Server Desktop Management. The basic idea is that all user directories live on a remote server. This allows anyone to log into any Apple Macintosh connected to the business or university network and have access to their home directory. The idea has been around since UNIX was developed. Unfortunately, when the age and design of your application pre-dates UNIX … okay, not really, but when the application wasn’t designed for this functionality you are guaranteed to run into a few problems along the way.
So why send me? Because I worked on the fix, silly, why else would you send such a shy and demure sort of guy who really doesn’t like people. (If you believe that I’ve got this bridge in Brooklyn that I just put on the market.) Seriously though, sending me all the way to Burbank to test a fix, and make me drive there to boot? What’s up with that?
The reason I went was because one of the known issues for enterprise and schools was the performance impact of running the Office applications when a user had a Remote Home Folder (RHF). Having spent my time working on emulators, video games and lots of other stuff that I can’t talk about, performance is my middle name. I was tasked, as well as a few other people with focusing on Remote Home Folder issues for this next version and getting things put right.
Reproducing the RHF bugs in our lab proved quite the challenge. The configurations we were trying to test with either differed in network bandwidth, network hardware, Mac OS software versions or even computer hardware versions. On top of that, some customers had a few users or 10,000 users. Trying to narrow down the configuration so we could reproduce the problem involved a great deal of creativity on our part. So when we have a customer close enough to our team that can try out our fixes it’s a great benefit to us and them!
So, back to the bug. Everyone out there that remembers Get1Resource and PBGetCatInfo, go get yourself a cookie … on me. Everyone who doesn’t know what those are, well I am about to explain why these two functions caused us a great deal of pain. Older applications for the Mac OS, think pre-OS X, used resource forks and resource files. These resource forks/files contained icons, data, code, window layouts, strings, etc. The way you could scan these files is by using the combination of these two old Apple APIs.
When Office launches, we scan all of the office applications, reading resource forks and collecting information to shove in the User Preferences for the suite, essentially building up caches. When you install Mac Office 2004, there are over 3000 files. On your local hard drive, this scan can take anywhere from 25 seconds to over a minute (depending completely on hardware). See where I’m going with this? Remember what I said about RHF and application design. Networks are not as fast as hard drives, no matter what anyone tells you, reading/scanning 3000+ files over the network is a bit of a slow thing to do. Some customers were reporting it would take up to 10 minutes to launch our application using RHF.
Now in Mac Office 2008, we have done a great deal of modernization across the board. No longer are we using resource forks and files. We have created application bundles, library frameworks and are completely Mach-O. Unfortunately with doing this, the number of files is now upwards of 20,000. That might seem like a lot, but considering that Safari and iTunes install about 4700 & 5800 files respectively, 20,000 files is not too bad for something as large as Office. (If you are interested in checking, try the following: Open Terminal; type cd /Applications; type find ./iTunes.app –name “*” | wc –l)
The reason behind this is that resource forks and files are now broken up into a single file per resource. Needless to say, if launching Office 2004 with RHF was slow it got much worse in Office 2008. If you were reading carefully, you would have noticed I said, “When Office launches.” Yeah, we discovered every time Office is launched it scans all the files, to verify the integrity of the caches it has build. That problem on top of the fact that we now had 20,000 files caused our performance to drop even further. This is where the team I work with started attacking the problem. It took a lot of digging, using Shark, our own internal debugging tools and really trying to understand what was going on, since the people who worked on this code I think left the company when ADA was created. After a great deal of work, many hours spent with stop watches and testers configuring lab machines, we were able to get the launch times down to 20 seconds or so depending on your machine, no matter if you are using RHF or not. Of course that was in our lab, which is why we needed to make a trip to a customer to test out all our hard work.
But why take a road trip? Burbank is a quick flight from our offices. Simply put, pregnant women don’t fly. My wife and I were expecting our first child and she had to be in San Diego for the week. Being the diligent husband, and actually really wanting to check out the San Diego office (which by the way doesn’t suck), I decided to drive down with her and along the way; I could stop by our valuable customer and verify our fix. This leads me back to why I drove all the way down to Burbank.
Watching the fix work in front of customers is a great feeling. I could say more about the visit, but there’s that implant in the back of my neck that hurts when I talk/type too much about things I’m not allowed to … ouch! They were quite happy with the results, and even happier that I was able to get another serious remote home folder bug fixed after I saw a problem they were having because we hadn’t seen it in our lab. That’s the great benefit of testing in a real world environment.
It was quite satisfying for me and the team: Stephen Shaw, Steven Splinter and Kirk Engelmeier to get this fixed for our customers.
Comments
Anonymous
November 02, 2007
Impressive improvement, but are you saying that we can expect any Office 2008 app to take around 20 seconds to launch under a normal setup (no RHF)? Is that just the first time any app in the Suite launches, with subsequent apps launching much faster, or every app launch is going to take 20 seconds or so? Because that seems pretty slow....Anonymous
November 03, 2007
The comment has been removedAnonymous
November 03, 2007
The launch time change is now only for the first time you launch and we build the preferences. I don't have exact numbers for the individual apps but it's definitely less than 20 seconds after you have built the caches for the first time. In Office 2004 we were always rebuilding the caches. After each launch. I could go into a great deal of detail about the San Diego office, but that's not the focus of this blog.Anonymous
November 03, 2007
THe question isn't time to open from NHD's, it's can you reliably WORK from them once the app is running. The fact that you can't reliably save to NHD's without bizarro workarounds that aren't reliable is far more of a problem. Really.Anonymous
November 04, 2007
Looking for testers? I have a 100mbs all switched network with 2500 RHFs on multiple servers using Workgroup Manager on all of them, run off all xServes and xServe Raids. Similar to what you are looking at when you test. My setup was build by several Apple badged field engineers using the best practices known. I am running 10.4.10 across the board on all machines. Lowest machine on the network is a G4 1.25ghz eMac with 512mb of RAM. Newest is a 1.83ghz iMac Intel based machine. How Office play in the managed network sandbox has been one of the thorns in my side for a long time..... Would LOVE to kick it around and provide feedback on it. Been using office in this kind of environment for over 10 years, well before OSX and been running it on OSX since early 2002 before Workgroup Manager first appeared.Anonymous
November 04, 2007
obligatory 'don't leave us in the dark about entourage' post.Anonymous
November 05, 2007
I just want entourage to fully support webdav, cookies (the standard installation of exchange now) and the outlook anywhere protocol.Anonymous
November 05, 2007
I think the business Mac community has lost faith of what Office 08 is going to be. With these illrelelvent posts, and no real info on Entourage, and the whole Office 08 platform; you can expect to see an increase on virtualizers and increase on Windows based Office apps. Its too bad that Microsoft doesn't want Macs in the enterprise, and defend a PC monopoly in which includes Intel Macs now. Microsoft always was behind the curve and was skeptical of every new technology (ie Intel Macs and Macs in the enterprise). I might just keep Fusion and my Office 03 on my Windows partition on my Mac.Anonymous
November 06, 2007
Ok I just realized that to answer a question about Messenger, a blog from over a year ago was referenced??? Please, can you at least update us on how that is going? Can we expect a new version of Messenger along with Office 08?Anonymous
November 06, 2007
VBA in Office 2008 - is it going to happen?Anonymous
November 06, 2007
What about Messenger Mac?????Anonymous
November 07, 2007
In the new PowerPoint, will you be able to see what you're typing when using a white font on a dark background? Lets be honest, as good as I'm sure many of the whizzy new features are, only a tiny percentage of people will ever actually use them. Whereas fixing this would help nearly everyone using PPT.Anonymous
November 08, 2007
wow, you guys are really bending over backwards to get that Office 2008 preview. Let me guess you are all busy writing code for exchange integration? Even better will Mac Office 2008 be 2xSuper Cool charged for Leopard? How about for OS 10.6 aka Tabby. Sorry just in a bad mood because I have been supporting entourage users whose meetings don't appear when they accept and the answer I get from MS support is "corrupted headers". Geez, thanks.Anonymous
November 12, 2007
The real question is what is the Sharepoint and PST support looking like? I have a gaggle of users waiting to update their machines. Expected Sharepoint and PST support are the two main selling points.Anonymous
November 14, 2007
And what about a patch for this issue in Office 2004? Surely it should be a simple switch, to turn off cache rebuilding every load--I do understand that there can be unanticipated ramifications that require significant testing, however. Frankly, some of us have and are jumping ship.Anonymous
November 24, 2007
abeness has a good point in the above post. Anyone at Microsoft reading? Or do you guys only like to hear yourself talk? Hello? Anyone home?Anonymous
November 29, 2007
You have bigger things to worry about than this! Six times is in the last week using Mac Office 2008 beta I have seen an 18-slide ppt file from Office 2003 with embedded Excel tables and graphs on almost each slide crash a Mac Pro running 10.5.1 so HARD that after a hard restart of the machine, the ppt file was COMPLETELY GONE from the HD. Nothing resembling the file name left on the HD or in the recovered folders file. Yesterday, a Word file from Office 2003—again with a lot of Excel tables—did the exact same thing. AND no, it is not the Mac, which is rock solid under all other uses and apps. If this has not been fixed in the next 6 weeks, MBU is in for some very sleepless nights in January.Anonymous
November 30, 2007
Yes it would be great to be able to roll the fix back into Office 2004 but it's not as simple. As I pointed out in the post, Office 2004 is completely resourced based (resource forks) and there are other differences between Office 2004 & Office 2008 that don't make it a trivial fix.