Problem - Kevin Tafuro

need to use one of the three ring3 serializing instructions (cpuid, iret, and rsm) to
invalidate the cache. This can be performed with a macro:
#define INVALIDATE_CACHE asm volatile( \
"pushad \n" \
"cpuid \n" \
"popad \n")
The pushad and popad instructions are needed because the cpuid instruction overwrites
the four general-purpose registers. Once again, as with the call to mprotect( ),
note that the use of the cpuid instruction is suspicious and will draw attention to the
code of the protection. It is better to place the call to the decrypted code far enough
away (16 bytes should be sufficient, because only 486 and Pentium CPUs will be
affected) from the actual decryption routine so that the decrypted code will not be in
the instruction cache.
This implementation of self-decrypting code is a simple one; it could be defeated by
pulling the decryption code from the binary, decrypting the protected code, then
replacing the call to the decryption routine with nop instructions. This is possible
because the size of the encrypted code is the same as the decrypted code; a more
robust solution would be to use a stronger encryption method or a compression
method, and extract the protected code to a dynamically allocated region of memory.
However, such a method requires extensive manipulation of the object files
before and after linking. You might consider using a commercially available binary
packer to reduce development and testing time.
See Also
Recipes 5.23, 12.3, 12.9
This is the Title of the Book, eMatter Edition
Copyright © 2007 O’Reilly & Associates, Inc. All rights reserved.
Using Self-Modifying Code | 699