Partager via

Integer Overflow and operator::new

  • Article

As Raymond Chen pointed out last year (https://blogs.msdn.com/oldnewthing/archive/2004/01/29/64389.aspx), there is a potential integer overflow when calling operator::new. The C++ compiler in Visual Studio 2005 automatically generates defensive code to mitigate this potential vulnerability.

 

Code like this:

 

class CFoo {

public:

      CFoo() {m_p = NULL;}

private:

      void *m_p;

};

void *func(size_t count) {

      if (!count) return NULL;

      return new CFoo[count];

}

 

When compiled with VS2005, yields output like this:

 

; _count$ = esi

; 14 : if (!count) return NULL;

  00000 85 f6 test esi, esi

  00002 74 36 je SHORT $LN4@func

; 15 : return new CFoo[count];

  00004 33 c9 xor ecx, ecx

  00006 8b c6 mov eax, esi

  00008 ba 04 00 00 00 mov edx, 4

  0000d f7 e2 mul edx

  0000f 0f 90 c1 seto cl

  00012 f7 d9 neg ecx

  00014 0b c8 or ecx, eax

  00016 51 push ecx

  00017 e8 00 00 00 00 call ??2@YAPAXI@Z ; operator new

 

As you can see (assuming you can read assembly code!) the multiply occurs (mul edx) and then the CL register is set or not set depending on the value if the overflow flag. Now here’s the funky part, ECX will end up being 0x00000000 or 0xFFFFFFFF, and because of the next operation (or ecx) the ECX register will either be 0xFFFFFFFF or the value held in EAX, which is result of the initial multiply. This is then passed to operator::new which will fail in the face of 2^N-1 by returning NULL or throwing an exception.

 

It’s better to fail, and potentially crash the application, than have an integer overflow that may lead to code execution.

 

Of course, the correct solution is get the code right in the first place by performing good arithmetic hygiene, and in this case, restrict the maximum number of objects you’ll create:

 

void *func(size_t count) {

      if (!count || count > MAX_FOO_ALLOWED) return NULL;

      return new CFoo[count];

}

Comments

  • Anonymous
    December 06, 2005
    The comment has been removed
  • Anonymous
    December 12, 2005
    So why not put an overflow check inside the implementation of new??

    I would argue that if new CFoo[(1<<31)+1] runs a pointer to an array of [1] allocated object, that's a bug in new, and should be fixed in the RTL.
  • Anonymous
    December 27, 2005
    The mul instruction is the slowest way to multiply the number by 4. You can get the same result faster with LEA edx, [edx * 4] or with SHL edx, 2 or with ADD edx, edx / ADD edx, edx

    What was optimization switches (/O1, /O2) when you compiled this code?
  • Anonymous
    January 04, 2006
    The comment has been removed
  • Anonymous
    January 04, 2006
    The comment has been removed
  • Anonymous
    June 04, 2006
    PingBack from http://codeka.com/blogs/index.php/dean/2006/06/05/integer_overflows_the_next_big_thing
  • Anonymous
    January 10, 2007
    MS07-004 does not affect Windows Vista, even though the coding bug is there. Why? The bug is an integer