audit

INELUCTABLE MODALITYOF LINUX AUDITA story of hate and delectationFaster & Smarter IR

AUDITD & AUDIT ARE MUTUALLYEXCLUSIVEWRONG!Wait for the din of shocked gasps to become quiescent, then proceed

AUDIT IS NOT MAGICALwhat is the meaning of this?audit_syscall_entryaudit_syscall_exit• Determines if the syscallshould be audited.• Initializes underlyingaudit_context structure fromthe current task_struct.• Emits several messages withdata associated with thesyscall over the netlink socket.• The last message is always oftype “AUDIT_EOE”

AUDIT IS NOT MAGICALsome various data which is logged at exit• return status• execve• sockaddrs• fd pairs• pid / auid / uid / sessionid• current working directories• path information

CAN’T STOP HERE, THISIS BAT COUNTRY

FEAR AND LOATHING IN KERNEL/AUDIT.Cthere can be only oneOnly one process can read from the audit netlink socketthis is a good thing - the kernel only has to maintain one bufferCreating a second reader will hijack the first and will not be restored on exitI get it, otherwise the kernel would be required to keep a backlog stack of processes

FEAR AND LOATHING IN KERNEL/AUDIT.Cdebugging is impossiblePrior to linux 3.8, when the audit backlog was hit, and audit_log_start wascalled during schedule_timeout : a deadlock would occur and fuck youso if you’re in gdb and hit a breakpoint, it was a raging race to disable auditI always forgot. The holes in the walls are a testament to that

FORMAT DISAPPROBATION :(The kernel is to blame for this shameful log format!audit_log_format(ab,"a0=%lx a1=%lx a2=%lx a3=%lx items=%d"" ppid=%d pid=%d auid=%u uid=%u gid=%u"" euid=%u suid=%u fsuid=%u"" egid=%u sgid=%u fsgid=%u tty=%s ses=%u",context->argv[0], context->argv[1], ...);but what is the alternative?An overly complex binarymessage format?How about this JSON thing I’vebeen hearing so much about?You’re insane.A JSON encoder in the kernel?You’re insane.

FORMAT APPROBATION :)Just follow these simple rules• The “serial number” is the kernel’s wayof designating multiple messages into• Everything that comes from the kernelis a key value pair, treat it like so.a single group. It is up to the user-landapplication to reassemble.• Unquoted values are (usually) deemedas “untrusted” strings, encoded as• User-land sourced messages arealways encapsulated in a key of “msg”ascii-hex.• If you were like me, stop bitching anddeal with it.

THINE ENEMY LIES WITHIN LEGACYAUDITING AUDITD• Performance problems under load.• Limited output format.• Difficult to extend.• Impossible to read.• Poorly designed (opinion).• Did I mention performance issues?• I’ve seen better code in openssl.

THINE ENEMY LIES WITHIN LEGACYThe mere presence of a commentcontaining “Global” is a good signthat the rest will be, in all probability,terrible.Is this what I think it is? Every singlemessage from the kernel is insertedinto a thread queue.!Also: “FIXME” in production code willalways induce cringe.

DEPRECATIONGOALS• Lower resource utilizationunder high load• Don’t reinvent the wheel, ifthe wheel isn’t broken• Extend (or create) logging• Abstract EVERYTHINGand filtering capabilities• Follow all of the rules in the• Keep some backwardsnext slide.compatibility with auditd

RULE ONE THROUGH ∞An afterthought in auditd.

LOW LEVEL PROCESSINGTHE AUDITD METHODLIBEV FOR NETWORK AND SIGNAL IOMANUAL NETLINK SOCKET HANDLINGOUR METHODLIBEVENT FOR NETWORK AND SIGNAL IOLIBMNL FOR NETLINK SOCKET HANDLING

!! WARNING !!No statements about how libev is faster than libevent.These comments are usually some variant ofregurgitated information based on the flawedperformance comparisons found on the libev website.“It’s only cheating if you do it on purpose.”

PROCESSING : LIBMNLcreating the socket and registering with the kernel

PROCESSING : RAW NETLINKcreating the socket and registering with the kernelthere is actually more

PROCESSING : LIBMNLreceiving a message from the netlink socketlibmnl does all the ugly work

PROCESSING : RAW NETLINKreceiving a message from the netlink socketthere is actually more

PROCESSING : MESSAGESdealing with the raw datapost processingruntime groupingThe method used by auditdrequiring an external application toparse and join multiple messagesusing the “serial number” as agrouping key.The method used by oursystem which appends datareceived from the kernel to a listuntil the final AUDIT_EOE packethas been processed.

PROCESSING : MESSAGE GROUPSan abstract example; executing “tail -f tsaudit.log”ungrouped- serial=43480, type=SYSCALL, syscall=“sys_execve”, exe=“/usr/bin/tail”- serial=43480, type=EXECVE, argc=2, a0=“tail”, a1=“tsaudit.log”- serial=43480, type=CWD, cwd=“/var/log”- serial=43480, type=PATH, name=“/usr/bin/tail”- serial=43480, type=EOEgrouped[]{“type" : "SYSCALL",“syscall" : "execve",“exe" : "/usr/bin/tail"},{“type" : "EXECVE",“argc" : 2,“argv” : [ “tail”, “tsaudit.log” ],},{“type" : "CWD",“cwd" : "/var/log"},{"type": "PATH","name": "/usr/bin/tail"}

PROCESSING : MESSAGE GROUPSA few more fun examples of grouping./var/resolvconf/interface$ cat eth0.dhclient nginx: int fd = open(“/www/index.html”); // fd == 13 fd = accept(“192.168.56.1:51997”); user “mthomas” started a pam session[]{"exe": "/bin/cat","comm": "cat","ses": 10,"auid": 4294967295,"pid": 31335,"ppid": 31334,"items": 2,"exit": 0,"success": "yes","syscall": "execve","epoch": 1399248110,"serial": 855516,"type": "SYSCALL"},{"a1": "eth0.dhclient","a0": "cat","argc": 2,"epoch": 1399248110,"type": "EXECVE"},{"cwd": "/run/resolvconf/interface","epoch": 1399248110,"type": "CWD"},{"name": "/bin/cat","epoch": 1399248110,"type": "PATH"}[]{"exe": "/usr/sbin/nginx","comm": "nginx","ses": 238,"pid": 966,"ppid": 965,"items": 1,"a3": "fffffffffffffffb","a2": 0,"a1": "800","a0": "ee7c05","exit": 13,"success": "yes","syscall": "open","epoch": 1392316421,"serial": 301316,"type": "SYSCALL"},{"cwd": "/","type": "CWD"},{"ogid": 0,"name": "/www/index.html","type": "PATH"}[]{"exe": "/usr/sbin/nginx","comm": "nginx","ses": 238,"pid": 966,"ppid": 965,"items": 0,"a3": "800","a2": "7fff8afba6cc","a1": "7fff8afba6d0","a0": 0,"exit": 12,"success": "yes","syscall": "accept4","epoch": 1392316421,"serial": 301314,"type": "SYSCALL"},{"saddr": "192.168.56.1","port": 51997,"prot": "ipv4","type": "SOCKADDR"}[]{}"res": "success","terminal": "ssh","addr": "192.168.56.1","hostname": "babby.local","exe": "/usr/sbin/sshd","acct": "mthomas","op": "PAM:session_open","ses": 24,"auid": 1000,"uid": 0,"pid": 10469,"epoch": 1393886985,"serial": 3393,"type": "USER_START"

PARSING“Every year, one out of ten programmers will commit suicide due tomaintaining parsers written in C”Every C developer who has had to maintain a parser in C

PARSING AUDIT MESSAGESyou be the judgetype=SYSCALL msg=audit(1386803107.182:7960575): arch=c000003e syscall=288 success=yes exit=26 a0=7a1=7fff986ec590 a2=7fff986ec58c a3=800 items=0 ppid=952 pid=956 auid=4294967295 uid=33 gid=33 euid=33 suid=33fsuid=33 egid=33 sgid=33 fsgid=33 ses=4294967295 tty=(none) comm="nginx" exe="/usr/sbin/nginx" key=(null)AUDITD METHODBRUTE FORCE~/Code/auditd$ egrep '(strstr|strchr|strtok|strcmp|strcasecmp|strdup|strcat|sprintf|snprintf)' auparse/*.c | wc -l448OUR METHODSTATE DRIVEN~/Code/tsaudit$ egrep '(strstr|strchr|strtok|strcmp|strcasecmp|strdup|strcat|sprintf|snprintf)' auparser/*.c | wc -l4

PARSING : BRUTE FORCEone of many horrible things you will encounter

PARSING : STATE DRIVENswitch / case generates lookup tables - faster

PARSING AUDIT MESSAGEStaking things to the next levelTURBO BOOSTING CONDITIONAL LOGIC- Generate a perfect hash tableusing “gperf”.- Add validation and auto-parsersfor specific key values.- Assign “known” keys and valuesto an enumerable type.- Determine if the value of a keycan be treated as a different datatype, such as an integer or- Filter out keys and values whichboolean.we deemed unnecessary forfurther processing

SO WE CAN DO STUFF LIKE THISwith perfect hash tables - lookups are O(1)

AND THIS

FILTERINGoptionally preprocess data before it is logged- Load per-instance or per-output LUA script duringstartup.- Convert the grouped messages to a native LUA table.- Call a pre-defined LUA function.- A non-zero return will drop the message.- A zero return will continue processing the message- If a LUA table is returned, it is converted to JSON andused as the output.

FILTERINGExample : simple boolean filter

FILTERINGExample : return a table which is converted to JSON on outputfunction find_set(k, set)for _,v in pairs(set) doif k == v thenreturn 1endendreturn 0end!!function tsaudit_filter(data)local ret = {}local comms = { 'irqbalance', 'whoopsie', 'top', 'dhclient' }!for k,ent in pairs(data) doif find_set(ent['comm'], comms) == 1 then-- if any of the keys are found, return this table which will be-- transformed into the JSON { "this" : "filtered", "dont" : "log" }ret = {this="filtered", dont="log" }end!if ent['success'] == 'no' then-- if the syscall did not succeed, then return 1 which will not generate a-- log.ret = 1endend!return retend

LOGGING : AUDITDOUTPUT TYPES1. File2. There is no 2Don’t worry, auditd can be extended with “audispd” plugins!

AUDISPD : A MONUMENTAL HACKnew term : “infinite noose”For each plugin, audispd will fork,execve, and send audit messagesto stdout.A plugin just has to have the abilityto read from stdin.if this design seems sane to you, keep it a secret, and start sharpening that programming knife

LOGGING : OUR WAY1. ZeroMQ2. SyslogOUTPUT TYPESas of right now3. Audit Emulator4. AIO5. Raw6. Nanomsgpluggable chained inlinedjson

ACCESSIONmeticulous attention to abstraction enhances creativityA fully functional auditd’esqueapplication can be written in under50 lines of C.• rtnetlink• netlink connector• netlink inet diag• netlink task statsIntroduced many other autonomousinputs which can be integratedseamlessly.• pcap data• userland audit

CEREBRATION•Additional methods for grouping relateddata, further reducing overhead.•Add simple analysis and statisticalgathering functionality.•Continue abstractions to avoid thepotential bloat that comes with featurecreep

COME WORK WITHUS@threatstack