Quick introduction to reverse engineering for beginners

fld QWORD PTR _a$[esp-4]
; current stack state: ST(0) = _a, ST(1) = _b
fcom ST(1) ; compare _a and ST(1) = (_b)
fnstsw ax
test ah, 65 ; 00000041H
jne SHORT $LN5@d_max
fstp ST(1) ; copy ST(0) to ST(1) and pop register, leave (_a) on top
; current stack state: ST(0) = _a
ret 0
$LN5@d_max:
fstp ST(0) ; copy ST(0) to ST(0) and pop register, leave (_b) on top
; current stack state: ST(0) = _b
ret 0
_d_max ENDP
FCOM is different from FCOMP is that sense that it just comparing values and leave FPU stack in the same
state. Unlike previous example, operands here in reversed order. And that is why result of comparision in
C3/C2/C0 will be different:
∙ If a>b in our example, then C3/C2/C0 bits will be set as: 0, 0, 0.
∙ If b>a, then bits will be set as: 0, 0, 1.
∙ If a=b, then bits will be set as: 1, 0, 0.
It can be said, test ah, 65 instruction just leave two bits — C3 и C0. Both will be zeroes if a>b: in
that case JNE jump will not be triggered. Then FSTP ST(1) is following — this instruction copies ST(0)
value into operand and popping one value from FPU stack. In other words, that instruction copies ST(0)
(where _a value now) into ST(1). After that, two values of _a are at the top of stack now. After that, one
value is popping. After that, ST(0) will contain _a and function is finished.
Conditional jump JNE is triggered in two cases: of b>a or a==b. ST(0) into ST(0) will be copied, it
is just like idle (NOP) operation, then one value is popping from stack and top of stack (ST(0)) will contain
what was in ST(1) before (that is _b). Then function finishes. That instruction used here probably because
FPU has no instruction to pop value from stack and not to store it anywhere.
Well, but it is still not over.
GCC 4.4.1
d_max proc near
b = qword ptr -10h
a = qword ptr -8
a_first_half = dword ptr 8
a_second_half = dword ptr 0Ch
b_first_half = dword ptr 10h
b_second_half = dword ptr 14h
push ebp
mov ebp, esp
42