Problem - Kevin Tafuro

Solution
Unfortunately, integer coercion and wrap-around problems currently require you to
be diligent.
Best practices for such problems require that you validate any coercion that takes
place. To do this, you need to understand the semantics of the library functions you
use well enough to know when they may implicitly cast data.
In addition, you should explicitly check for cases where integer data may wrap
around. It is particularly important to perform wrap-around checks immediately
before using data.
Discussion
Integer type problems are often quite subtle. As a result, they are very difficult to
avoid and very difficult to catch unless you are exceedingly careful. There are several
different ways that these problems can manifest themselves, but they always boil
down to a type mismatch. In the following subsections, we’ll illustrate the various
classes of integer type errors with examples.
Signed-to-unsigned coercion
Many API functions take only positive values, and programmers often take advantage
of that fact. For example, consider the following code excerpt:
if (x < MAX_SIZE) {
if (!(ptr = (unsigned char *)malloc(x))) abort( );
} else {
/* Handle the error condition ... */
}
We might test against MAX_SIZE to protect against denial of service problems where
an attacker causes us to allocate a large amount of memory. At first glance, the previous
code seems to protect against that. Indeed, some people will worry about what
happens in the case where someone tries to malloc( ) a negative number of bytes.
It turns out that malloc( )’s argument is of type size_t, which is an unsigned type. As
a result, any negative numbers are converted to positive numbers. Therefore, we do
not have to worry about allocating a negative number of bytes; it cannot happen.
However, the previous code may still not work correctly. The key to its correct operation
is the data type of x. Ifx is some signed data type, such as an int, and is a negative
value, we will end up allocating a large amount of data. For example, if an
attacker manages to set x to –1, the call to malloc( ) will try to allocate 4,294,967,295
bytes on most platforms, because the hexadecimal value of that number
(0xFFFFFFF) is the same hexadecimal representation of a signed 32-bit –1.
Preventing Integer Coercion and Wrap-Around Problems | 89
This is the Title of the Book, eMatter Edition
Copyright © 2007 O’Reilly & Associates, Inc. All rights reserved.