Share via

More on testing Mp3 files and the boundary testing debate

  • Article

Over the past 3 days I have learned more about Mp3 file encoding and decoding than I have since the technology was introduced. I don't spend time downloading files from the Internet to burn CD's, I don't own an iPod or Mp3 player, or a digital video recorder. So, prior to this I haven't really paid attention to this technology and was quite ignorant of the various tools available and their capabilities. But, I must say it is pretty fascinating from a technology standpoint even though I am not an audiophile or videophile.

I still disagree with Pradeep's assertion regarding boundary testing and the notion of no fixed boundaries, but respect Pradeep's expertise in the area of Mp3 technology. An Aussie gentleman by the name of Dean Harding pointed out my incorrect assumption regarding bitrate encodings and explained the LAME encoder does allow a freeformat option in "expert" mode to produce a fixed bitrate in one kilo bit increments between 8 kb/s and 640 kb/s. (Thanks for serving up the pie Dean.) However, of 30+ common decoders I only discovered 4 decoders supported freeformatted Mp3 files even if the encoded bitrate is less than 320 kb/s. Only one decoder (WinAmp MAD) is capable of decoding files above 600 kb/s.

So, (other than me having to eat a big helping of humble pie) where does that leave us in the specific debate about boundary testing, and Pradeep's question "As a tester have you ever seen a boundary?" To that, I shall adamantly reply "yes" there are specifc boundary conditions in software. Some are easy to find, some are not so easy. A tester's ability to correctly identify a boundary value are heavily influenced by his/her in-depth domain and 'system' knowledge. For example, using the knowledge of Mp3 encodings I have learned over the past 3 days let's go back and review what tests I would design based on Pradeep's original description of the audio decoder that played an Mp3 file within the range of 24 kb/s to 196 kb/s.

Since 196 kb/s is not a standard Mp3 encoding supported by ISO standards let's assume the Mp3 player used either a Cdex, LAME, I3dec, or WinAmp MAD decoder. Using this as a reference, and some recently acquired domain knowledge I would design a set of initial tests using the following sample test data (files encoded with the specified criteria).

  1. 23, 24, 25 kb/s - Specified minimum value and minimum -1, and minimum +1 values to analyze relational operators used to artificially constrain the encoding range to a low end of 24 kb/s.
  2. 195, 196, 197 kb/s - Specified maximum value and maximum -1 and maximum + 1 values to anlyze to analyze relational operators used to artificially constrain the encoding range to a high end of 196 kb/s.
  3. 16 kb/s - this is the next ISO standard encoding bitrate below the specified minimum, so although the decoder does not support a file encoded at 23 kb/s (min -1 value) I would still want to check at the next lower standard value.
  4. 224 kb/s - this is the next ISO standard encoding bitrate above the specified minimum (same reason as explained above.)
  5. 32, 40, 48, 56, 64, 80, 96, 112, 128, 144 (see #6) , 160, 176, 192 kb/s - these are the typical ISO standard Mp3 encodings within the specified range, so we should assume that all these must work properly because there is a high probability of decoding files using these bitrates. Since there are not many of them test each one.
  6. 143, 144, 145 kb/s because the 144 kb/s bitrate seems to be an interesting value that "sticks out" more than others, and so I may also want to analyze the values around that particular value for any other anomalies
  7. Generate several randomly encoded files in the following ranges (between 24 kb/s and 127 kb/s) (between 128 and 143 kb/s), and (between 145 kb/s and 196 kb/s) to gain confidence the decoder can decode non-standard encoded files within the specified range without having to test all 174 or so possibilities

These are not the only tests I would execute; however, they would be the first set of tests I would design and execute to make sure the code at least does what it is supposed to do. Any failure in the above cases means our basic program functionality has some serious flaws. Once I established the program does what it is supposed to do (including handling expected errors gracefully), then I would begin exploring other possibilities including rigorous falsification/negative testing.

Pradeep indicated a file encoded at 96 and 128 kb/s crashed the system. These are not boundaries conditions (unless the developer did something totally unreasonable), and unfortunately since we can only assume files encoded above and below 96 and 128 kb/s played correctly we will never really know the cause of this problem (unless Pradeep did some root cause analysis and will share those findings). However, a failure with 128 kb/s is really a red flag to me because this happens to be the most prevalent bitrate for encoding Mp3 files. As a tester I would really want to know why unit testing or build acceptance testing, etc. didn't at least hit the most probable encoding format (the happy path) before throwing crap code over the wall for Pradeep to test.

I hope the reader takes away a few lessons from all this (besides the obvious one of not going off half cocked especially if you lack expertise in the specific context (e.g. Mp3 encodings)). For example, 

  • In-depth knowledge of the domain space (including the data set, how the data is encoded, and how the data set can and cannot be manipulated in code both correctly and incorrectly), industry standards, and how the domain space interacts with the system are critical for greater test effectiveness
  • The less we know about the domain, the data set, the system interaction the less effective are the application of specific techniques to identify specific classes of defects
  • Boundary testing is simply one technique (systematic procedure to solve one type of a complex problem). The boundary value analysis technique is designed to identify a specific class of defects involving incorrectly specified constant values, incorrect use of data types, or casting between data types, artificially constraining data types, and incorrect usage of relational operators. It is not effective to identify other classes of defects.
  • Boundary conditions simply do not exist at the extreme physical ranges, there could also be multiple boundary conditions within the overall range. (A good example is the Unicode repetoire. The Unicode BMP spans from U+0000 to U+FFFF, but within this range there are several important boundaries one must take into consideration when using Unicode data depending on the purpose of the test and the application under test (e.g. Private use area, surrogate pair area, etc).)
  • Understanding how to decompose the data set into equivalence class partition subsets exposes boundary conditions we might not otherwise consider
  • There is a great deal of detail in the code that can expose interesting information for a tester
  • When talking technology, be as specific and precise as possible to avoid ambiguity
  • And perhaps most importantly, one technique or one approach to testing is not sufficient. As testers we must gather and learn to use a great variety of skills and knowledge and approach the problem from multiple perspectives to be most effective in our roles.

OK...now time to get some really bitter coffee to wash down that humble pie :-)

Comments

  • Anonymous
    April 22, 2007
    Hi William, This has been your third post in continuation to the the long and scattered discussions about BVA over various blogs and corresponding comments, mainly invloving you, Pradeep and Shrini. That has been really an interesting reading. It was especialy good to see you take back your words (which you thought were a bit rude for Pradeep) and then take back the situation based on nothing but logic. I am impressed. You have discussed the subject in detail. I did not know anything about MP3 encoding and decoding, your posts made me know and grasp some details. What matters to me the most is that all this has clarified certain base concepts, which are generic and are deeply associated to the profession of software testing. I have especially liked your note on the "lessons" in the current post. With all your permission, I would like to refer this and some other content in your posts (ofcourse with links ponting to them), so that more and more people get to know this piece of knowlede. Frankly speaking, you, Pradeep and Shrini are talking about the same subject and have almost the same notion. All of you say that BVA is not a dumb job. It requires analysis. All of you say that the boundaries are not fixed. I found that you have contradictions, but not many. Most of the times, it was not about way that you all think about BVA, it was all about the language you used while talking about BVA - just a game of words. The above paragraph is just to end a hidden clash between three testing experts whom I like. I believe that all the discussion on BVA could have been done in a much healthier manner. Sometimes, it puts the readers in a bit embarassing situation, because I am sure that we like you all for your knowledge. I hope all my words are taken in a positive way. Regards, Rahul Verma.