Quick introduction to reverse engineering for beginners

1.2 Stack
Stack — is one of the most fundamental things in computer science. 6 .
Technically, this is just memory block in process memory + ESP register as a pointer within this block.
Most frequently used stack access instructions are PUSH and POP. PUSH subtracting ESP by 4 and then
writing contents of its sole operand to the memory address pointing by ESP.
POP is reverse operation: get a data from memory pointing by ESP and then add 4 to ESP. Of course, this
is for 32-bit environment. 8 will be here instead of 4 in x64 environment.
After stack allocation, ESP pointing to the end of stack. PUSH increasing ESP, and POP decreasing. The
end of stack is actually at the beginning of allocated for stack memory block. It seems strange, but it is so.
What stack is used for?
1.2.1 Save return address where function should return control after execution
While calling another function by CALL instruction, the address of point exactly after CALL instruction is
saved to stack, and then unconditional jump to the address from CALL operand is executed.
CALL is PUSH address_after_call / JMP operand instructions pair equivalent.
RET is fetching value from stack and jump to it — it is POP tmp / JMP tmp instructions pair equivalent.
Stack overflow is simple, just run eternal recursion:
void f()
{
f();
};
MSVC 2008 reporting about problem:
c:\tmp6>cl ss.cpp /Fass.asm
Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 15.00.21022.08 for 80x86
Copyright (C) Microsoft Corporation. All rights reserved.
ss.cpp
c:\tmp6\ss.cpp(4) : warning C4717: ’f’ : recursive on all control paths, function will
cause runtime stack overflow
... but generates right code anyway:
?f@@YAXXZ PROC ; f
; File c:\tmp6\ss.cpp
; Line 2
push ebp
mov ebp, esp
; Line 3
call ?f@@YAXXZ ; f
; Line 4
pop ebp
ret 0
?f@@YAXXZ ENDP ; f
... Also, if we turn on optimization (/Ox option), the optimized code will not overflow stack, but will
work correctly 7 :
?f@@YAXXZ PROC ; f
; File c:\tmp6\ss.cpp
; Line 2
6 http://en.wikipedia.org/wiki/Call_stack
7 irony here
4