Quick introduction to reverse engineering for beginners

#include
#include
void main()
{
int handle;
};
We got:
handle=open ("file", O_RDWR | O_CREAT);
public main
main proc near
var_20 = dword ptr -20h
var_1C = dword ptr -1Ch
var_4 = dword ptr -4
push ebp
mov ebp, esp
and esp, 0FFFFFFF0h
sub esp, 20h
mov [esp+20h+var_1C], 42h
mov [esp+20h+var_20], offset aFile ; "file"
call _open
mov [esp+20h+var_4], eax
leave
retn
main endp
Let’s take a look into open() function in libc.so.6 library, but there is only syscall calling:
.text:000BE69B mov edx, [esp+4+mode] ; mode
.text:000BE69F mov ecx, [esp+4+flags] ; flags
.text:000BE6A3 mov ebx, [esp+4+filename] ; filename
.text:000BE6A7 mov eax, 5
.text:000BE6AC int 80h ; LINUX - sys_open
So, open() bit fields are probably checked somewhere in Linux kernel.
Of course, it is easily to download both Glibc and Linux kernel source code, but we are interesting to
understand the matter without it.
So, as of Linux 2.6, when sys_open syscall is called, control eventually passed into do_sys_open kernel
function. From there — to do_filp_open() function (this function located in kernel source tree in the file
fs/namei.c).
Important note. Aside from usual passing arguments via stack, there are also method to pass some of
them via registers. This is also called fastcall 2.5.3. This works faster, because CPU not needed to access a
stack in memory to read argument values. GCC has option regparm 46 , and it’s possible to set a number of
arguments which might be passed via registers.
Linux 2.6 kernel compiled with -mregparm=3 option 47 48 .
What it means to us, the first 3 arguments will be passed via EAX, EDX and ECX registers, the other ones
via stack. Of course, if arguments number is less than 3, only part of registers will be used.
46 http://ohse.de/uwe/articles/gcc-attributes.html#func-regparm
47 http://kernelnewbies.org/Linux_2_6_20#head-042c62f290834eb1fe0a1942bbf5bb9a4accbc8f
48 See also arch\x86\include\asm\calling.h file in kernel tree
55