Refactoring and performance
I'm currently reading Test Driven Development in Microsoft .NET. There's a lot of stuff in the book I like, but there was one thing that rubbed me the wrong way. In the "Refactoring by example" chapter, they take a prime number algorithm and refactor it. In one step, the algorithm loops up to sqrt(n). They merge the loop with one that loops up to n. They do some hand waving that this might not be as efficient, but you should check to make sure. This seems to go against the philosophy of refactoring they describe up to the point of the book. Refactoring should improve the internal structure of the code without making externally visible changes. And, the test driven philosophy is that you write tests for all externally visible features. So, shouldn't there be tests that define the performance requirements and shouldn't the verification that the performance is still acceptable after a refactoring be done by verifying that a test still passes?
Of course, this is easier said than done. Automated performance testing is hard, especially on arbitrary machines. In the MacBU they have a great performance testing infrastructure that tests the performance of every build. This tells you if any changes were made that hurt performance, or how well any performance improvements worked. This was always done on a specific machine on an isolated network to keep results reproducible. Doing tests on every developer's machine is a bit more complex. How do you know how long a test should take? What do you compare it to? Even defining performance requirements can be difficult. What CPU speed? How much memory? How many other applications running? etc.
I wish the book addressed this issue. I don't know if any other books in this genre do...
Comments
- Anonymous
October 23, 2004
yes Automated performance testing IS hard. It's my day job and i hate it. Reproduceability of results is near impossible, measurement machine has to be reset to a predefined state before every measurement (usually the original pristine state). This includes OS, cache(s) and filesystem. Now the hard part is not in the operations, but trying to model the variance of the results across runs and understanding how timing effects can affect scores.
Maybe Rico Mariani can enlighten us on how it's done in the .net group? - Anonymous
October 23, 2004
Refactoring is not about making the code run faster, it's about making the code readable for others and of course for your self. To make the software easier to understand, you often make changes that will cause the program to run more slowly. Refactoring could make the software go more slowly, but it also makes the software more amenable to performance tuning. Changes that will improve performance usually make the program harder to work with. Hardware is cheap, but resources are expensive. In long-terms it's sometimes more important to make the code reusable and readable than making the code optimized to run some extra milliseconds faster. When you have used refacorring for a while, you will se how easily it will be to optimize the code to run fast when the code is easy to understand and read. Code that other can understand will make the code more maintainable. - Anonymous
October 24, 2004
I agree refactoring is not about making the code run faster, but you can't just randomly make things slower just to make the code more readable. I believe that you should have automated tests of performance when making changes like this just as you should have automated tests to make sure you still get the same output. If you make lots of changes that make your app slower and slower as you go along and plan performance tuning later, there's a good chance you won't get to that tuning. As with any flaw, the sooner you fix it, the cheaper it is. - Anonymous
October 24, 2004
Dan:<br><br>I can see your point and I also think it could sometime be important to run a performance test. If I know that my refactoring could affect performance badly, I should consider if I really need to do the refactoring or not. If I knew that the code should be reused by other domain objects in my domain model, I should do the refactoring, event if it could make my software run slower. But there is always a big "But!". Sometimes, performance could be more important (which is something we probably know when we are designing our domain model) than reusing code or make the code more understandable.<br><br>>"If you make lots of changes that make your app slower and slower as you go along and plan performance tuning later, there's a good chance you won't get to that tuning. As with any flaw, the sooner you fix it, the cheaper it is." <br><br>If you need to make the optimization early, which could be useful in some cases, the readability and maintainability of the code could be affected, and in long-term, it could be cheaper to maintain the software than it was to do the performance optimization later in the project. But it’s always as but here also. I have been in a project that we know the performance was important, even though we did a lot of refactoring that affected the performance for making the code more readable and maintainable. At the end of the project we spend some time with optimization of the code, and the optimization didn’t took long, because the code was so well written and easy to maintain, so the optimization was easy to do. But this doesn’t mean that every project will run that smoothly. - Anonymous
October 24, 2004
I understand your points and have seen lots of cases of premature optimization causing problems. However, I want to make a couple of points.
First, you write "If I know that my refactoring could affect performance badly..." How do you know if your refactoring could affect performance badly without tests? To do refactoring with confidence, you need an automated test to be sure.
Second, I wanted to clarify that when I wrote "the sooner you fix it, the cheaper it is", I didn't mean that you should improve performance early. What I meant is that if you make a change that negatively affects performance (beyond what's acceptable), the sooner you find this problem, the cheaper it is to fix. - Anonymous
October 24, 2004
The comment has been removed - Anonymous
October 24, 2004
I think there are only 20 percent of code which contribute to 80 percent of the running time and hence are key for performance.
And the rest 80% do not affect the performance much. But if we can make them more readable and maintainable then I think we are better off than optimizing them for performance. So we should focus our refactoring effort on those 80% of code (But do not totally ignore the rest 20%)
And sometimes refactored code are also more efficient. One example is using database pooling. - Anonymous
October 24, 2004
>"How do you know if your refactoring could affect performance badly without tests? To do refactoring with confidence, you need an automated test to be sure".
By looking at the code, you can see if it will affect performance badly. But as you pointed out, some operations would need a performance test before we know if it will affect the performance badly. Having a well-factored program helps with optimization. Because it gives you time to spend on performance tuning. If you have a well-factored code, you can add function more quickly. This gives you more time to focus on performance. With a well-factored program you have a finer granularity for your performance analysis. The profiler you use will take a look at smaller parts of the code, which are easier to tune. Code that is cleaner, will give you a better understanding of your options and of what kind of tuning will work. Reafactoring will give us slower software, but will make it easier to tune during the optimization.
TDD is not about testing, it's more about developing code that works in the way you want it to work. So I should first make sure my code works before and after I have done some refactoring. If I notice with a profiler that the code will not perform well, I will do the optimization (probably fairly late in development), and after the optimization I will run the test again to see if the code works.
Maybe I'm out of the topic right now, but I like to share my thoughts, and are more interesting in others thoughts. - Anonymous
October 24, 2004
This has been an interesting discussion.
In my experience, when you have a mature product and are working on a new version, if you ignore performance, things will get slower and slower over time if you ignore performance. If you only focus on performance at the end, you'll have a lot of work ahead of you just to get back to where you were when you started. Just like it's important to not introduce bugs as you do feature work or refactoring, it's important not to introduce changes that degrade performance.
I do not believe the statement "By looking at the code, you can see if it will affect performance badly." You could also say "By looking at code, you can see if it will cause bugs." You can't. You need automated tests because we make mistakes. I also think working on making an application well factored, ignoring performance so you can have more time to focus on performance later is a mistake.
Fredrik writes that TDD is about "developing code that works the way you want it to work". I agree, and I think performance is part of that. Working correctly isn't just about the results, but about performance too.
Christian noted that it's hard to specify performance requirements on the unit level. I agree, that's difficult. You need higher level automated tests for performance if you want to refactor with confidence.
Manas mentions that only 20% of code contributes to 80% of performance. I agree there too. However, if you are changing a random piece of code in an application, how do you know whether or not it's in the 20% that matters or the 80% that doesn't? Experience shows that you can't know - this is the whole reason that premature optimization is a problem. You need automated performance testing to know whether or not a change has a unintended performance consequence. I've seen changes that are seemingly unrelated to performance have disastrous effects, and the sooner you catch these, the better. - Anonymous
October 24, 2004
About:
>"By looking at the code, you can see if it will affect performance badly"
My point was that, you can see if some code could affect performance like splitting a for loop into two separated loops with are iterating through the same data but uses to different operations etc. It was that kind of “code” I mend, and as I also wrote, there are some operations we can’t know how it will affect performance and in that case we can’t only look at the code. So small pieces of code that we will refactoring to make it easier to read, could affect performance and by looking at the code we can now if the new code after rafactoring could decrease the performance. Here is another example that could clarify what I mend: When you are open a connection against a data base without connection pooling, the connection will take time, maybe you write code where you want to reuse the same connection for two operation against a data source for saving the extra time it will take two open twp connections, but when you have done some refactoring you have made two connections and use tow methods where each operation is now located in its on methods. This could affect performance and that is something you can see by looking at the code. - Anonymous
October 25, 2004
In my experience with new project development, by FAR the biggest concern with initial versions is.. uh.. not crashing. Everything else, including perf, is tertiary to that overriding goal. It really doesn't matter if we crash 50% faster!
Now, after 2-3 releases, we start to really nail performance. It's a lot easier to attack the performance problem after you've got the basics down, and you can profile the code a bit to get to the 20% that is causing the performance problem.
Understandability and reliability are FAR, FAR more important than performance and need to be prioritized appropriately. So I'm not sure your "out of the gate" goal is realistic or even worth pursuing, honestly. - Anonymous
June 09, 2009
PingBack from http://greenteafatburner.info/story.php?id=3844