The Pit of Success
The
Pit of Success: in stark contrast to a summit, a
peak, or a journey across a desert to find victory through many trials and
surprises, we want our customers to simply fall into winning practices by using
our platform and frameworks. To
the extent that we make it easy to get into trouble we fail.
-Rico Mariani, MS Research MindSwap Oct 2003.
-----------------
I had a chance hear Rico Mariani do
his stump speech on performance of managed code to an audience of very senior
technical folks (not sure why they let me in)… As a performance architect on the CLR
team Rico has a ton of passion for how we can change internal CLR details to
make performance (more specifically workingset) better. He talked about some very cool things we
are doing in Whidbey around NGen, VTable layout etc for saving a few bytes per
type or instance. All very cool
stuff. And in fact we see some
fairly substantial performance wins in our performance test cases in the
lab.
But our experience with in house,
real world applications has been that they are not realizing this level of
performance win. Why? Turns out their performance is dominated
by other factors. The big wins we
realized at the CLR level are just noise compared to other performance problems
in the applications. With just a
few days of work our pref team was able to improve the performance of one of
these in house applications more significantly than all the CLR level
improvements combined. Their findings are
published here
. This is NOT because the app developers
are a bunch of clowns. Rather it is
because, as hard as we tried in V1, there were still some places where the
design of the platform leads them down the wrong path.
Because he was talking (mainly) to a
set of platform folks he admonished us to think about how we can build platforms
that lead developers to write great, high performance code such that developers
just fall into doing the “right thing”.
Rico called this the Pit of Success. That concept really resonated with
me. More generalized, it is the key
point of good API design. We should
build APIs that steer and point developers in the right direction. Types should be defined with a clear
contact that communicates effectively how they are to be used (and how not
to). I am not just talking about
the docs and samples (although those are good) but in the design of the
APIs. For example, give the
“pretty” name to the types most developers should use (ie. “Foo” and
“FooBase”).
A powerful thought, crystallized
well…. Enjoy.
Comments
Anonymous
October 03, 2003
I definitely agree with your comments about API names; it always seemed unfortunate to me that there is a class called BindingManagerBase...I really would have preferred this to be just BindingManager. My rule of thumb is that I only use a 'Base' suffix in a class name only when clients arent using the type in a polymorphic fashion. So CollectionBase is fine, but BindingManagerBase is not...Of course if there had been an IBindingManager then this would not have been a problem...Anonymous
October 03, 2003
Seems like a lot of great words that don't carry much meaning.Pretty much every language/API/framework developer would like to have things "just work", such that the obvious way to write code was the best.So I don't see a great deal of value of stating that, it should be obvious. Concrete ways of achieving this is what the big issue is. I'd love to see more and deeper usability research done on programming languages and frameworks, which might give us some data to base such work on.Otherwise, when you design something such that the "obvious" way is the right way, what you're really doing is designing it such that the right way is the way that's obvious to you. Which is a start, but not necessarily what's good for other people.It's funny how there seems to be an industry push towards usability guidelines backed up by studies and customer feedback when it comes to the UI in the end product. Yet the end product that us programmers use day in and day out is really designed on a "seems like the right thing to do to me" basis. Something that's largely been rejected due to poor results for the end users, yet we're made to suffer through it!Anonymous
October 03, 2003
Shane, your point is taken. In fact we have several fulltime API usability engineer that are constantly developing and running studies about what make good API design. There is nothing like sitting on the other side of the one-way class watching a developer struggle to use an API you thought you designed well. We are working hard to give that experience to designers of as many of the API sets as we can. But on a more leveraged scale are factoring much of that information into the Design Guidelines document and then into the design of our APIs. It seems there is clearly some value in getting more of the raw usability information out… I will see what I can do.Anonymous
October 03, 2003
Although this is tangentially related, I wanted to add a "real life" experience. At my current day job, we're developing an extensive WinForms based desktop withing an n-tier .Net app. Our target deployment platforms is Win2k machines @500 Mhz w/256Mb ram (I would bet that this platform is the norm for large corporations such as the one I work for.) We recently started Ngen’ing on deploy, and the performance difference is incredible (for the better). From casual observation using perfmon, it appears that code is being JITed “all of the time” – possibly re-JITed or something else. The JITing is taking up most of the CPU most of the time on these low-end machines. Of course we will figure out the exact story of what and why this is so with enough time and effort, and correct the bad thing(s) we’ve done code-wise so that Ngen won’t make such a difference.My point is that I shouldn’t be in this situation. I’ve been a C/C++ guy for many years, and it’s my experience that you have to try very hard to not do the right thing to get bad performance from a C++ implementation. The same should hold true for .Net related activities.Please, throw me into the pit.Anonymous
October 03, 2003
Phil, I have to disagree with you on 'trying very hard to not to do the right thing' with respect to C++ implementation. Recently, I had to review a piece of C++ code which had been passing large vectors by value to various methods. Since the guy who wrote the code no longer works here, I don't know if he was just missing const &, or had no idea of how objects are passed by value in C++ (and how time-expensive memory allocation is).Anonymous
October 05, 2003
Brad,link to gotdotnet page you have in the post is not working can you repost the correct linkAnonymous
October 05, 2003
Sorry about that -- fixedAnonymous
October 06, 2003
I like the metaphor... especially coming from a CLR guy ;) At first read (this blog and the headtrax perfwhacking session) I can see how you can do some of this with the CLR. It's a little less clear as you move up the food chain. I hope you find the following of interest... here's a half step up the food chain.Jitrz (java/.net) can do some interesting things beyond loading/compiling. Jitrz do late-bound, for lack of proper term, optimizations... things that take into account the run-time/access behavior... things that cannot be known at compile time (static). They establish an internal feedback loop regarding code and use.Now take the Headtrax perfwhacking report... much attention was paid to assemblies, disk access, etc.... even suggestions on optimizing deployment assemblies based on run-time specifics vs the logical model. Gah! [note: i understand why] I'd like the CLR (et. al.) to do for my assembly loading what the Jitrz do for late-bound optimization. In other words... figure it out and fix it. Ha! It has all the info on access patterns and probably load times.So in my perfect pit, headtrax developers too, the assembly performance (load) and access patterns would be analyzed and assemblies "regrouped" to improve performance. I can keep my [il]logical assembly groupings... the CLR optimizes it over time (based on it's current implementation and how users "use" the software)... and users get good performance by accident... I mean fall into the Pit of Success./jhdAnonymous
October 09, 2003
The perf advantages of JIT compiling mentioned by John Dhom seem only to apply to server side applications. For client side apps, the user is so annoyed by the JIT delays that the JIT compiler cannot do much of any optimization.Consider that the new release of Office is not written in managed code. From the blog of Jason Zander: "We did some experiments early on in the Runtime as proof of concept for our managed C++ compiler which included recompiling Word as an IL image. It worked great! But it was slow. Office is a big application, and using the JIT for this case didn't put our best foot forward."MS cannot eat their own dogfood, but it is on our plates. The JIT compiler simply does not make sense for client side apps. Nor does it make sense for tiny devices such as the PocketPC. I can understand how small bits of code downloaded from the internet need JIT compiling for security. But to compile every installed client app, every time it is run, and to throw away the results: such a wasteful process (by default) throws us into the pit of failure! Also loading the IL when only the native code and metadata is needed, in an ngened app, makes no sense. Why does MS release and recommend technology that is not good enough for its own applications?Anonymous
October 14, 2003
The comment has been removedAnonymous
October 26, 2003
Thank you for the response, anonymous. I was not only focusing on big client apps. If you look at the perf and windows forms newsgroups you will see a common complaint: the old app, ported to .net, starts up too slow. Often the problem is the jit on the designer generated code. Another common problem is the number of and size of dlls loaded. Even small apps often start up slowly compared to older C++ versions.Is the solution profiling, possibly ngen, and reworking the app? This is the only solution currently, but it is contrary to the "pit of success" idea at the top of this page.I don't believe there must be a trade-off. We should be able to use managed code and not have to think about performance any more than with a C++ app -- that is, focus on algorithms and data structures, not jit compile time. If something is jit once, it should be saved, and reused next time. Or ngen should be the default for desktop apps. Managed code is a great thing, but if the perf problems are not addressed, for client apps it may gradually aquire a reputation similar to that of Java client side code: slow and fat.I think managed code has the potential to replace almost all application code. Not all OS code, or games, but almost everywhere C or C++ is used. A real focus on perfomance and the idea of the "pit of success", by MS, would take it in that direction.Anonymous
May 08, 2006
My team is starting detailed design and implementation work over the next week or so and I know of few...Anonymous
August 05, 2006
I recently had a fun time baking cookies with my three year old son.. He had a great time scooping out...Anonymous
August 15, 2006
转自 http://www.hanselman.com/blog/Anonymous
August 15, 2006
转自http://www.hanselman.com/blog/Anonymous
September 08, 2006
PingBack from http://www.sashasydoruk.com/?p=30Anonymous
July 23, 2007
Most software projects fail . But that doesn't mean yours has to. The first question you should askAnonymous
September 15, 2007
Eric Lippert notes the perils of programming in C++ : I often think of C++ as my own personal Pit ofAnonymous
September 16, 2007
PingBack from http://aoortic.com/?p=10011Anonymous
September 16, 2007
PingBack from http://aoortic.com/?p=10024Anonymous
September 16, 2007
PingBack from http://aoortic.com/?p=10038Anonymous
September 17, 2007
PingBack from http://projects.akela.ro/portal/architectsblog/?p=104Anonymous
September 20, 2007
PingBack from http://gloryfish.org/a-broken-billboard/Anonymous
October 16, 2007
PingBack from http://www.brokenbuild.com/blog/2007/10/16/notes-from-alistair-cockburns-talk-at-the-seattle-apln-meeting/Anonymous
November 25, 2007
PingBack from http://coderpeon.ovh.org/?p=41Anonymous
January 23, 2008
PingBack from http://rowansimpson.com/2008/01/24/garr-reynolds-wisdom-part-i/Anonymous
April 02, 2008
When you design your API, make it easy to use and hard to misuse. Brad Abrams quotes Rico Mariani onAnonymous
April 24, 2008
When you design your API, make it easy to use and hard to misuse. Brad Abrams quotes Rico Mariani onAnonymous
May 20, 2008
PingBack from http://www.hanselman.com/blog/TheWeeklySourceCode27SuckLessLibraries.aspxAnonymous
May 20, 2008
I've been getting more and more interested in how folks extend their applications using plugins and things.Anonymous
May 22, 2008
The comment has been removedAnonymous
July 07, 2008
PingBack from http://www.ageektrapped.com/blog/the-missing-net-6-version-tolerant-serialization/Anonymous
September 28, 2008
PingBack from http://joshrobb.com/blog/2007/09/29/aspnet-usability-disaster/Anonymous
October 05, 2008
Yesterday Oren announced the release of RhinoMocks 3.5 .  While the RC version has been out forAnonymous
October 07, 2008
Yesterday Oren announced the release of RhinoMocks 3.5 .  While the RC version has been out forAnonymous
November 23, 2008
PingBack from http://journal.stuffwithstuff.com/2008/11/23/constrained-compound-design-pattern/Anonymous
January 11, 2009
One of my pet peeve questions I often see on various tech mailing lists is “How can I prevent situationAnonymous
February 05, 2009
One of the features I was really excited about for the MVC RC was the template/Model based scaffoldingAnonymous
March 19, 2009
Note: This article is submitted by Damon Payne for Silverlight: Write and Win contest .Thanks a lot,Anonymous
May 04, 2009
PingBack from http://www.elilopian.com/2009/04/27/managing-the-support/Anonymous
June 02, 2009
PingBack from http://indoorgrillsrecipes.info/story.php?id=10504Anonymous
June 07, 2009
PingBack from http://greenteafatburner.info/story.php?id=1227Anonymous
June 08, 2009
PingBack from http://toenailfungusite.info/story.php?id=3365Anonymous
June 15, 2009
PingBack from http://einternetmarketingtools.info/story.php?id=21746