2013-12-05_tcpflow-and-BE-update

TCPBEflowTCP analysis with tcpflow;packet carving with bulk_extractorSimson L. GarfinkelNaval Postgraduate SchoolDecember 5, 2013

This talk presents new carving and analysis features intcpflow and bulk_extractor.tcpflow 1.4 — Turns PCAP files into “transcripts”pcap file(s)TCPflowHTTP headers & contentsmtp email messagesshNetBIOSunprocessed packetsbulk_extractor 1.4 — a feature extractor for disk images.optimisticdecodinglots of stuffThese tools frequently find information that’s invisible to other tools.2

tcpflow is a command-line tool for processing streams.Wireshark and EtherPeek look at packets—These tools do not handle millions of packets or TCP connectionsOmniPeekWiresharkGUI-based stream processing tools can be hard to automateNetworkMiner3

tcpflow can run batch or live-captureBatch operation is typical forensics:filename1.pcapfilename2.pcapfilename3.pcapfile.pcap.rarfile.pcap.gzfile.pcap.zipTCPflowconnection1connection1-HTTP-file.jpegconnection2connection3connection4...Live-capture is useful for testing & stunts• Run with ‘-c’ console output to see content of TCP connections as they go by.• Output to file system and read ‘report.xml’ or use alert_fd and process each file as it isclosed.4

tcpflow turns each side of the TCP connection into a file.These files are then processed with standard tools.tcpflow splits 1-N pcap files into multiple TCP connections.• Each connection is put in two files (client ➜ server / server ➜ client)• HTTP sessions are decompressed & explodedTCPflowExample:$ tcpflow -a -o out-nitroba -r nitroba.pcap$ ls -l out-nitroba3648 Jul 22 2008 004.071.104.187.00080-192.168.015.004.35950462 Jul 22 2008 004.078.212.029.00080-192.168.015.004.35458138 Jul 22 2008 004.078.212.029.00080-192.168.015.004.35458-HTTPBODY-001.html462 Jul 22 2008 004.078.212.029.00080-192.168.015.004.35712138 Jul 22 2008 004.078.212.029.00080-192.168.015.004.35712-HTTPBODY-001.html136520 Jul 22 2008 008.012.217.125.00080-192.168.015.004.328222550 Jul 22 2008 008.012.217.125.00080-192.168.015.004.32822-HTTPBODY-001.gif1582 Jul 22 2008 008.012.217.125.00080-192.168.015.004.32822-HTTPBODY-002.gif58123 Jul 22 2008 008.012.217.125.00080-192.168.015.004.32822-HTTPBODY-003.gif...# of bytessent5

tcpflow turns each side of the TCP connection into a file.These files are then processed with standard tools.tcpflow splits 1-N pcap files into multiple TCP connections.• Each connection is put in two files (client ➜ server / server ➜ client)• HTTP sessions are decompressed & explodedTCPflowExample:$ tcpflow -a -o out-nitroba -r nitroba.pcap$ ls -l out-nitroba3648 Jul 22 2008 004.071.104.187.00080-192.168.015.004.35950462 Jul 22 2008 004.078.212.029.00080-192.168.015.004.35458138 Jul 22 2008 004.078.212.029.00080-192.168.015.004.35458-HTTPBODY-001.html462 Jul 22 2008 004.078.212.029.00080-192.168.015.004.35712138 Jul 22 2008 004.078.212.029.00080-192.168.015.004.35712-HTTPBODY-001.html136520 Jul 22 2008 008.012.217.125.00080-192.168.015.004.328222550 Jul 22 2008 008.012.217.125.00080-192.168.015.004.32822-HTTPBODY-001.gif1582 Jul 22 2008 008.012.217.125.00080-192.168.015.004.32822-HTTPBODY-002.gif58123 Jul 22 2008 008.012.217.125.00080-192.168.015.004.32822-HTTPBODY-003.gif...# of bytessentDatesent5

tcpflow turns each side of the TCP connection into a file.These files are then processed with standard tools.tcpflow splits 1-N pcap files into multiple TCP connections.• Each connection is put in two files (client ➜ server / server ➜ client)• HTTP sessions are decompressed & explodedTCPflowExample:$ tcpflow -a -o out-nitroba -r nitroba.pcap$ ls -l out-nitroba3648 Jul 22 2008 004.071.104.187.00080-192.168.015.004.35950462 Jul 22 2008 004.078.212.029.00080-192.168.015.004.35458138 Jul 22 2008 004.078.212.029.00080-192.168.015.004.35458-HTTPBODY-001.html462 Jul 22 2008 004.078.212.029.00080-192.168.015.004.35712138 Jul 22 2008 004.078.212.029.00080-192.168.015.004.35712-HTTPBODY-001.html136520 Jul 22 2008 008.012.217.125.00080-192.168.015.004.328222550 Jul 22 2008 008.012.217.125.00080-192.168.015.004.32822-HTTPBODY-001.gif1582 Jul 22 2008 008.012.217.125.00080-192.168.015.004.32822-HTTPBODY-002.gif58123 Jul 22 2008 008.012.217.125.00080-192.168.015.004.32822-HTTPBODY-003.gif...# of bytessentDatesentSourceIP5

tcpflow turns each side of the TCP connection into a file.These files are then processed with standard tools.tcpflow splits 1-N pcap files into multiple TCP connections.• Each connection is put in two files (client ➜ server / server ➜ client)• HTTP sessions are decompressed & explodedTCPflowExample:$ tcpflow -a -o out-nitroba -r nitroba.pcap$ ls -l out-nitroba3648 Jul 22 2008 004.071.104.187.00080-192.168.015.004.35950462 Jul 22 2008 004.078.212.029.00080-192.168.015.004.35458138 Jul 22 2008 004.078.212.029.00080-192.168.015.004.35458-HTTPBODY-001.html462 Jul 22 2008 004.078.212.029.00080-192.168.015.004.35712138 Jul 22 2008 004.078.212.029.00080-192.168.015.004.35712-HTTPBODY-001.html136520 Jul 22 2008 008.012.217.125.00080-192.168.015.004.328222550 Jul 22 2008 008.012.217.125.00080-192.168.015.004.32822-HTTPBODY-001.gif1582 Jul 22 2008 008.012.217.125.00080-192.168.015.004.32822-HTTPBODY-002.gif58123 Jul 22 2008 008.012.217.125.00080-192.168.015.004.32822-HTTPBODY-003.gif...# of bytessentDatesentSourceIPSourcePort5

tcpflow turns each side of the TCP connection into a file.These files are then processed with standard tools.tcpflow splits 1-N pcap files into multiple TCP connections.• Each connection is put in two files (client ➜ server / server ➜ client)• HTTP sessions are decompressed & explodedTCPflowExample:$ tcpflow -a -o out-nitroba -r nitroba.pcap$ ls -l out-nitroba3648 Jul 22 2008 004.071.104.187.00080-192.168.015.004.35950462 Jul 22 2008 004.078.212.029.00080-192.168.015.004.35458138 Jul 22 2008 004.078.212.029.00080-192.168.015.004.35458-HTTPBODY-001.html462 Jul 22 2008 004.078.212.029.00080-192.168.015.004.35712138 Jul 22 2008 004.078.212.029.00080-192.168.015.004.35712-HTTPBODY-001.html136520 Jul 22 2008 008.012.217.125.00080-192.168.015.004.328222550 Jul 22 2008 008.012.217.125.00080-192.168.015.004.32822-HTTPBODY-001.gif1582 Jul 22 2008 008.012.217.125.00080-192.168.015.004.32822-HTTPBODY-002.gif58123 Jul 22 2008 008.012.217.125.00080-192.168.015.004.32822-HTTPBODY-003.gif...# of bytessentDatesentSourceIPSourcePortDestIP5

tcpflow turns each side of the TCP connection into a file.These files are then processed with standard tools.tcpflow splits 1-N pcap files into multiple TCP connections.• Each connection is put in two files (client ➜ server / server ➜ client)• HTTP sessions are decompressed & explodedTCPflowExample:$ tcpflow -a -o out-nitroba -r nitroba.pcap$ ls -l out-nitroba3648 Jul 22 2008 004.071.104.187.00080-192.168.015.004.35950462 Jul 22 2008 004.078.212.029.00080-192.168.015.004.35458138 Jul 22 2008 004.078.212.029.00080-192.168.015.004.35458-HTTPBODY-001.html462 Jul 22 2008 004.078.212.029.00080-192.168.015.004.35712138 Jul 22 2008 004.078.212.029.00080-192.168.015.004.35712-HTTPBODY-001.html136520 Jul 22 2008 008.012.217.125.00080-192.168.015.004.328222550 Jul 22 2008 008.012.217.125.00080-192.168.015.004.32822-HTTPBODY-001.gif1582 Jul 22 2008 008.012.217.125.00080-192.168.015.004.32822-HTTPBODY-002.gif58123 Jul 22 2008 008.012.217.125.00080-192.168.015.004.32822-HTTPBODY-003.gif...# of bytessentDatesentSourceIPSourcePortDestIPDestPort5

tcpflow turns each side of the TCP connection into a file.These files are then processed with standard tools.tcpflow splits 1-N pcap files into multiple TCP connections.• Each connection is put in two files (client ➜ server / server ➜ client)• HTTP sessions are decompressed & explodedTCPflowExample:$ tcpflow -a -o out-nitroba -r nitroba.pcap$ ls -l out-nitroba3648 Jul 22 2008 004.071.104.187.00080-192.168.015.004.35950462 Jul 22 2008 004.078.212.029.00080-192.168.015.004.35458138 Jul 22 2008 004.078.212.029.00080-192.168.015.004.35458-HTTPBODY-001.html462 Jul 22 2008 004.078.212.029.00080-192.168.015.004.35712138 Jul 22 2008 004.078.212.029.00080-192.168.015.004.35712-HTTPBODY-001.html136520 Jul 22 2008 008.012.217.125.00080-192.168.015.004.328222550 Jul 22 2008 008.012.217.125.00080-192.168.015.004.32822-HTTPBODY-001.gif1582 Jul 22 2008 008.012.217.125.00080-192.168.015.004.32822-HTTPBODY-002.gif58123 Jul 22 2008 008.012.217.125.00080-192.168.015.004.32822-HTTPBODY-003.gif...# of bytessentDatesentSourceIPSourcePortDestIPDestPortHTTPContent5

The filename is created from a template.Default template: %A.%a-%B.%b%V%v%C%cTemplate elements:%A/%a - source IP address/port%B/%b - dest IP address/port%E/%e - source/dest Ethernet Mac address%V/%v - VLAN number, '--' if no vlan/'' if no vlan%T/%t - Timestamp in ISO8601 format/unix time_t%c - connection_count for connections>0%C - 'c' if connection_count >0%N - (connection_number ) % 1000%K - (connection_number / 1000) % 1000%M - (connection_number / 1000000) % 1000%G - (connection_number / 1000000000) % 1000%% - Output a '%'TCPflow6

tcpflow automatically bins connections into directoriesDefault is no binning — good for less than 1,000 connections.Specify “-Fk” to bin by “connection number”• Prepends “%K/” to template$ ls -lFtotal 1821666606 Nov 26 08:04 000/60112 Nov 26 08:04 001/79628 Nov 26 08:04 002/66232 Nov 26 08:04 003/0 Nov 26 08:04 alerts.txt19076 Nov 26 08:04 report.pdf18631436 Nov 26 08:04 report.xml$ ls -lF out-nitroba-Fk 000/*3648 Jul 22 2008 000/004.071.104.187.00080-192.168.015.004.35950462 Jul 22 2008 000/004.078.212.029.00080-192.168.015.004.35458138 Jul 22 2008 000/004.078.212.029.00080-192.168.015.004.35458-HTTPBODY-001.html462 Jul 22 2008 000/004.078.212.029.00080-192.168.015.004.35712138 Jul 22 2008 000/004.078.212.029.00080-192.168.015.004.35712-HTTPBODY-001.html136520 Jul 22 2008 000/008.012.217.125.00080-192.168.015.004.328222550 Jul 22 2008 000/008.012.217.125.00080-192.168.015.004.32822-HTTPBODY-001.gif1582 Jul 22 2008 000/008.012.217.125.00080-192.168.015.004.32822-HTTPBODY-002.gif58123 Jul 22 2008 000/008.012.217.125.00080-192.168.015.004.32822-HTTPBODY-003.gif7

You can bin by IP address, port #, etc.For example, to bin by source IP address:$ tcpflow -T%A/%a-%B.%b%V%v%C%c -a -o out3 -r nitroba.pcap$ ls -lF out3 | headtotal 17864102 Nov 26 08:11 004.071.104.187/204 Nov 26 08:11 004.078.212.029/1768 Nov 26 08:11 008.012.217.125/136 Nov 26 08:11 008.012.221.123/442 Nov 26 08:11 012.129.147.065/136 Nov 26 08:11 012.129.210.041/204 Nov 26 08:11 012.129.210.046/578 Nov 26 08:11 012.130.060.002/102 Nov 26 08:11 012.130.081.249/$—Note 1: “file size” and “mod time” are not meaningful for directories.—Note 2: When binning by address or port, the two sides end in a different directories.8

eport.xml is a DFXML file that records what tcpflow did.There are five sections:• tcpflow build environment (OS, host, compiler, libraries)• tcpflow execution environment (OS, host, input, command options, etc.)• Specific configuration options (e.g. tdelta)• Each stream that was created• Information collected at end of run• CPU & memory required by the run9

Provenance makes it easier to use rapidly developing tools.4.2.1 (4.2.1 Compatible Apple LLVM 5.0(clang-500.2.79))2013-11-23T15:54:13Darwin13.0.0Darwin Kernel Version 13.0.0: Thu Sep 1922:22:27 PDT 2013; root:xnu-2422.1.72~6/RELEASE_X86_64Mucha.localx86_64src/tcpflow -Fk -a -o out-home2 -rpackets.pcap5022013-11-24T04:47:25Z10

tags generated by tcpflow match fiwalk’s.Note:031.013.069.160.00443-010.002.107.009.5213376a1e0610d76af60000e8a0710a6bb59c6• Processing flows with XML is non-standard but relatively easy.• 10-20GB DFXML files can be rapidly processed with dfxml.py (SAX-based parser).Other output options in development:• sqlite• Netflow? RFC5665 IPFIX? SiLK? FlowCollector?11

Decoded HTTP objects are represented with and recurisve s.......-HTTPBODY-001.gif3964...HTTPBODY-001.gif5048...12

Harassment atNITROBAState University

The caseNITROBAState UniversityYou are a staff member at the Nitroba University Incident Response Team.Lily Tuckrige is teaching chemistry CHEM109 this summer at NSU.Tuckrige has been receiving harassing email at her personal email address.• Tuckrige's personal email is lilytuckrige@yahoo.com• She thinks that it is from one of the students in her class.Tuckrige contacted IT support.• She sent a screen shot of one of the harassing email messages.• She wants to know who is doing it.istockphoto.com

To find out what's going on,Nitroba's IT sets up a packet snifferNITROBAState UniversityWeb ClusterNetworkSwitchLogging HostDesktop PCBut multiple people were using the same IP address.Solution: use browser finger printing.

The attack email:NITROBAState University

How to solve this problem:NITROBAState University1. Map out the Nitroba dorm room network.2. Find who sent email to lilytuckrige@yahoo.com• Look for a TCP flow that includes the hostile message• Find information that can tie that message to a particular web browser.3. Identify the other TCP connections that below to the attacker4. Find information in one of those TCP connections that IDs the attacker.

To solve the Nitroba University Harassment scenarioRun tcpflow on nitroba.pcap:$ tcpflow -r nitroba.pcap -o out$ ls -l out | cut -c 30- | head3648 Jul 22 2008 004.071.104.187.00080-192.168.015.004.35950462 Jul 22 2008 004.078.212.029.00080-192.168.015.004.35458462 Jul 22 2008 004.078.212.029.00080-192.168.015.004.35712136520 Jul 21 2008 008.012.217.125.00080-192.168.015.004.3282223943 Jul 21 2008 008.012.217.125.00080-192.168.015.004.3282417995 Jul 21 2008 008.012.217.125.00080-192.168.015.004.3282820064 Jul 21 2008 008.012.217.125.00080-192.168.015.004.32830879 Jul 21 2008 008.012.221.123.00080-192.168.015.004.33298213 Jul 21 2008 012.129.147.065.00080-192.168.001.064.34023...$18

Slides omitted

TCPflowInside tcpflow

tcpflow addresses important “real-world” capture problems.When we capture in the real-world, we see:• 1-sided captures (from asymmetric routing)• Missing packets (no SYN)• Duplicated packets• Out-of-order delivery• Hostile packetstcpflow handles these situations.24

tcpflow understands packets so you don’t have to.We think of TCP as a stream-based protocol:HTTP Request“GET /hello.txt HTTP/1.1”HTTP responseHTTP headersHTML contentBut connections are made up of packets:1SYN :80:58221 ACK234SYN/ACK :80“GET /hello.txt HTTP/1.1”DATA :80TCP 3 wayhandshake:58221resp5:58221“Hello World!”FIN67FIN :8025

26tcpflow makes transcripts from each flow.A flow is an (Source-IP, Source-Port, Dest-IP, Dest-Port) 4-tuple1SYN3SYN/ACKtcpflow“GET /hello.txt HTTP/1.1”4DATA7FIN2ACK5resptcpflow“hello world!”6FIN…

27Packets sometimes get delivered out-of-order.tcpflow doesn’t care.7FIN3SYN/ACKtcpflow“GET /hello.txt HTTP/1.1”1SYN4DATA6FIN5resptcpflow“hello world!”2ACK…

28tcpflow makes transcripts with missing SYNs & ACKs.1SYN3SYN/ACKtcpflow“GET /hello.txt HTTP/1.1”4DATA7FIN2ACK5resptcpflow“hello world!”6FIN…

29tcpflow will detect replay/rewriting accidents/attacks.2ACKw: 105respw: 11“hello world!”5resp2w: 11tcpflow“Mary had a little lamb, her fleece was white as snow.”5resp3w: 11“I cannot forecast to you the action of Russia.It is a riddle wrapped inside a mystery inside an enigma”6FIN…

30tcpflow will detect and split gigabyte-sized gaps in the flow.2ACKw: 105respw: 11“hello world!”tcpflow5resp3w: 1000000000 “hello world!”6FINw: 1000000013…

tcpflow can be extended and automated.Read files or report.xml after run for post-processing.• report.xml uses same DFXML as bulk_extractor & fiwalkReal-time use:• Watch report.xml (flushed when each connection is closed)• Create a pipe and read it with alert_fd (notified on each file open & close)cmd = "tcpflow -a -Fk -b 10485760 -S http_alert_fd=1 "+\"-i eth1 -o outdir tcp port 80"tcpflow_process =subprocess.Popen(shlex.split(cmd),stdout=subprocess.PIPE,stderr=subprocess.STDOUT)—Then watch stdout from tcpflow_process for files opened & closed.31

tcpflow 1.4 uses the bulk_extractor plug-in APIBasic structure of tcpflow similar to basic structure of bulk_extractor.• Expanded bulk_extractor API to handle call-backs for packets• Visualization handled as another module• Currently single-threadedTCPcreate_fileMD5packet iteratorHTTP & gzipOther bulk_extractor modules can be employed:• JSON• BASE64• GPS32

tcpflow — performance and comparisonData set: MIT Lincoln Labs ID99 EvaluationDataset:Week 2, Tuesday, InsideNumber of Packets 1585120Size of PCAP file:400,104,805 bytesNumber of TCP connections: 110,978tcpflow 1.4 Wireshark 1.10.3Time to process(Mac Pro)91.1 seconds 276 secondsTime to access eachTCP stream

SSL decryption issuesOptions for decrypting SSL:• Man-in-the-middle proxy (e.g. mitmproxy)• Decryption with private keyWireshark — decrypts SSL with private key.tcpflow — Currently no SSL decryptionNo SSL decryption with perfect forward secrecy (PFS) protocols• Generally authenticated Diffie-Hellman34

Presently there is poor browser support for PFS.“The actual cipher suites usedwhen connecting to 2.4 MillionSSL sites with the cipher suitesettings extracted from eachbrowser. *Opera does not include itsTLS 1.2 cipher suites.“SSL: Intercepted today, decrypted tomorrow”—Netcraft September 2013http://news.netcraft.com/archives/2013/06/25/ssl-intercepted-today-decryptedtomorrow.html35

The actual cipher suites used when connecting to 2.4 Million SSL sites with the cipher suite settings extracted from each browser. *Opera does not includeits TLS 1.2 cipher suites.nternet Explorer does particularly poorly as it does not support any cipher suite that uses both RSA public keys and non-elliptic-curve DH key exchange,Currently PFS is not much of an issue.hich includes the most popular PFS cipher suite. The PFS cipher suites that IE does support have a lower priority than some of the most commonlyupported non-PFS cipher suites. Curiously, IE does support DHE-DSS-AES256-SHA, which uses the rarer DSS authentication method, but not the veryopular DHE-RSA-AES256-SHA.Browser priority Cipher Suite Real-world usage in SSL Survey1 AES128-SHA 63.52%2 AES256-SHA 2.21%3 RC4-SHA 17.12%4 DES-CBC3-SHA 0.41%5 ECDHE-RSA-AES128-SHA 0.08%6 ECDHE-RSA-AES256-SHA 0.21%7 ECDHE-ECDSA-AES128-SHA 0.00%8 ECDHE-ECDSA-AES256-SHA 0.00%9 DHE-DSS-AES128-SHA 0.00%10 DHE-DSS-AES256-SHA 0.00%11 EDH-DSS-DES-CBC3-SHA 0.00%12 RC4-MD5 16.46%Internet Explorer 10's cipher suite ordering and the actual negotiated cipher suite in Netcraft's SSL survey. PFS cipher suites are highlighted in bold andgreen.afari supports many PFS cipher suites but non-elliptic-curve cipher suites are used only as a last resort. As several non-PFS ciphers have a higher priority,“SSL: Intercepted today, decrypted tomorrow”eb servers respecting the browser's preferences will end up selecting a non-PFS cipher suite even if the web server itself does support some (non ellipticurve)PFS cipher suites.hrome, Firefox, and Opera all do better, preferring PFS cipher suites ahead of non-PFS at any given strength level — for example Opera's preference list—Netcraft September 2013http://news.netcraft.com/archives/2013/06/25/ssl-intercepted-today-decryptedtomorrow.htmltarts: DHE-RSA-AES256-SHA, DHE-DSS-AES256-SHA, AES256-SHA, DHE-RSA-AES128-SHA, DHE-DSS-AES128-SHA, AES128-SHA. Netcraft did notnclude any cipher suites only present in TLS 1.2 which includes many of Opera's PFS cipher suites, so the results for Opera form a lower bound on theumber of SSL sites using PFS with Opera.one of the browsers change their user interface perceptibly to reflect the presence of PFS akin to the way EV certificates are treated to a green addressar. Google Chrome and Opera show the cipher suite used (in popups or dialog boxes), but they rely on a user understanding the implications of wordinguch as "[..] ECDHE_RSA as the key exchange mechanism".eb server support for PFS36

One-page visualization of packet flows - NitrobaTCPFLOW 1.4.0b1Input: /corp/nps/packets/2008-nitroba/nitroba.pcapGenerated: 2013-06-16 21:32:36Date range: 2008-07-21 18:51:07 -- 2008-07-21 23:13:47Packets analyzed: 91,144 (55.02 MB)Transports: IPv4 100%6 MB100%5 MB3 MB2 MB0 MB54 00 06 12 18 24 30 36 42 48 54 00 06 12 18 24 30 36 42 48 54 00 06 12 18 24 30 36 42 48 54 00 06 12 18 24 30 36 42 48 54 00 06 124 hours, 22 minutes (3 minute intervals)0%HTTP HTTPS Port 52227 MB0 B192.168.8.0/21208.111.148.674.125.0.0/2069.22.167.21569.22.167.21469.22.167.224/27196.0.0.0/6204.0.0.0/8Top Source Addresses66.192.0.0/10209.16.0.0/12100%0%40 MB0 B192.168.8.0/21196.0.0.0/6192.168.1.64239.255.255.25069.22.167.21566.192.0.0/10216.112.0.0/1569.22.167.224/28Top Destination Addresses208.111.148.669.22.167.214100%0%1) 192.168.8.0/21 - 6.56 MB (11%) 1) 192.168.8.0/21 - 39.58 MB (71%)2) 208.111.148.6 - 5.41 MB (9%) 2) 192.168.1.64 - 7.77 MB (14%)3) 74.125.0.0/20 - 4.55 MB (8%) 3) 239.255.255.250 - 818.61 KB (1%)45 MB80100%6 MB803971033148100%0 B443397103299233148522234884Top Source Ports3488235804347420%0 B35956348823488433944Top Destination Ports3574234118352640%1) 80 - 44.78 MB (84%) 1) 80 - 5.85 MB (11%)2) 443 - 1.82 MB (3%) 2) 39710 - 5.41 MB (10%)3) 39710 - 178.85 KB (0%) 3) 33148 - 4.44 MB (8%)37

808023One-page visualization of packet flows - MITLL ID99TCPFLOW 1.4.0Input: /corp/mitll/packets/ideval99/week2/tuesday/inside.tcpdumpGenerated: 2013-11-27 18:42:42Date range: 1999-03-09 08:00:01 -- 1999-03-10 03:03:20Packets analyzed: 1,571,748 (373.65 MB)Transports: IPv4 100%13 MB100%9 MB6 MB3 MB0 MB12 36 00 24 48 12 36 00 24 48 12 36 00 24 48 12 36 00 24 48 12 36 00 24 48 12 36 00 24 48 12 36 00 24 48 12 36 00 24 48 12 36 00 24 48 12 36 0019 hours, 3 minutes (12 minute intervals)0%FTP SSH Port 23 Port 25 HTTP HTTPS Port 12417 Port 2477716 MB172.16.112.194172.16.112.100206.128.0.0/9209.67.29.11172.16.114.148205.176.0.0/13172.16.114.50208.0.0.0/8136.0.0.0/6207.25.71.141100%25 MB172.16.117.64/26172.16.113.105172.16.113.204172.16.112.207172.16.112.194172.16.116.44172.16.114.207172.16.116.128/25172.16.114.148172.16.113.84100%0 BTop Source Addresses0%0 BTop Destination Addresses0%1) 172.16.112.194 - 16.15 MB (4%) 1) 172.16.117.64/26 - 24.91 MB (6%)2) 172.16.112.100 - 15.86 MB (4%) 2) 172.16.113.105 - 22.98 MB (6%)3) 206.128.0.0/9 - 12.54 MB (3%) 3) 172.16.113.204 - 22.69 MB (6%)240 MB100%30 MB100%0 B232022252477712417Top Source Ports2188217377132910%0 B252224604749628262Top Destination Ports194401548189330%1) 80 - 239.63 MB (65%) 1) 80 - 30.44 MB (8%)2) 23 - 26.32 MB (7%) 2) 23 - 23.06 MB (6%)3) 20 - 23.17 MB (6%) 3) 25 - 8.05 MB (2%)38

tcpflow 1.5 — wifi supportwifi support provided by Wifipcap• Open source C++ from CMU 2003—Jeffrey Pang — Radiotap implementation—Doug Madory — Wifi Parser• Implement packet handlers by subclassinga C++ callback class.tcpflow supports*:• radiotap parsing• 802.11 parsingTCPflow—* supports means that 80% of the work is done.39

Radiotap information available at TCP levelPer packet information includes:• channel• FHSS (Frequency Hopping Spread Spectrum) information• Rate• Signal dbm• Noise dbm• Quality• tx attenuation• tx power• flags• retriesTCPflowMany applications!40

Radiotap packet capture is built into MacOSThe legality of clicking the“Start” button is unclear.41

802.11 link-layer information available at TCP levelEach 802.11 frame has 3 or 4 MACs• Destination• Source• [Access Point MAC] or [transmitter & receiver]TCPflowtcpflow can:• Produce map of SSIDs to MACs• Label each flow with the BSSID MAC, single strength, etc.42

eport.xml will contain a list 802.11 beacons:(MACs, SSIDs, and counts)43

plot_wifi_aps.py will transform this into a ‘dot’ output$ python plot_wifi_aps.py report.xml > report.dot$ circo -Tpdf report.dot > report.pdf00:17:13:8a:44:d700:15:13:64:42:13 linksys00:0c:13:28:43:c3Janice00:19:13:00:4d:ac00:02:13:8d:47:57HOTEL00:15:13:64:4f:be00:02:13:d7:45:79Michael00:14:13:ce:48:84 Hunter 00:14:13:ce:48:8500:02:13:5c:40:8cSuper SpfiZALMOD-SEC00:19:13:4b:48:6dZALMOD00:0c:13:4f:43:3944

tcpflow — future improvementsBuffer connections in RAM — FasterIncorporate ssldump — Decrypt SSL connections (for a while)Improved visualizations — Useful in the fieldWireshark integration• Lots of people are using Wireshark• Use tcpflow to re-implement Wireshark’s “follow TCP connection”• Or use Wireshark’s ‘dissectors’ in Wireshark.45

BEbulk_extractorimprovements

ulk_extractor finds more information, faster than before.Version 1.4 significantly improves transformations & feature recognitionNew decoders & extractors:• RAR (extraction & carving)• Encrypted RAR (carving)• ZIP (extraction & carving)• JPEG carving• XOR (experimental)optimisticdecodingUsefuldata“Optimistic” means that every decoder checks EVERY BYTE.47

zip.txt — potential zipfile headersZIP has become the defacto archive format.• zip, jar, docx, pptx, etc.• ZIP64 provides for files larger than 4GiB• Allows faster access to components that .tar.gzFOOBARFOOBARzipfilestructurebulk_extractor finds “local file headers.”LocalHeaderDatalocal file header signature 4 bytes (0x04034b50)version needed to extract 2 bytesgeneral purpose bit flag 2 bytescompression method2 byteslast mod file time2 byteslast mod file date2 bytescrc-324 bytes validation for accuracy!compressed size4 bytesuncompressed size4 bytesfile name length2 bytesextra field length2 bytesfile name(variable size)extra field(variable size)CentralDirectory49

zip.txt decodes every potential header of every zip archive# Filename: charlie-2009-12-11.E01# Feature-Recorder: zip# Feature-File-Version: 1.162865144 000024.tif 000024.tif2088002009-11-19T16:57:0000decompressedNotice:• Original file name & mtime are recoveredYou can identify & extract fragments from:• MSOffice and OpenOffice documents• Java programs; Android & iPhone applications50

Beverly, Robert, Simson Garfinkel and Greg Cardwell, "Forensic Carving of Network Packets andAssociated Data Structures", DFRWS 2011, Aug. 1-3, 2011, New Orleans, LA. BEST PAPER AWARDPacket carving withoptimistic decompression52

We hypothesized that binary network data structures werepresent on non-volatile media.“Binary network data structures” includes:• Network packets• Memory structures associated with network connectionsWe know that binary data are present in RAM.CPUAre they present on storage?Mechanisms:• Swap files• Hibernation files• Binary data structures stored in files.RAMDisk/FlashPC or Phone53

Are low-level binary network data structures persisted tostorage?Approach: use bulk_extractor to find the packets.e.g.:We created scan_net.• Searches for—BPF “pcap” packet headers—IPv4 & IPv6 packets—Ethernet headers—Windows Socket Structuresstruct ip {u_int ip_v:4, /* version */ip_hl:4; /* header length */u_char ip_tos; /* type of service */u_short ip_len; /* total length */u_short ip_id; /* identification */u_short ip_off; /* fragment offset field */u_char ip_ttl; /* time to live */u_char ip_p; /* protocol */u_short ip_sum; /* checksum */struct in_addr ip_src,ip_dst; /* source and dest address */}Ran with bulk_extractor• Processed 1817 disk images• Recorded found IP addressesin feature files.R. Beverly, S. Garfinkel, G. Cardwell (NPS) Network Carvinginputscan_netIP addresses54

Author's personal copy84Binary carving found information not in ASCIIOf the 1817 images:cross all operating systems, we are always able to retrieve the• binary IP addresses: 723 drives (≈40%)achine’s local IP address. Table 4 shows the precision ofetwork carving using only high-confidence and all addresses.• Average text IP addresses per drive: 2258e label an IP address as a true positive when it is discovered• Average binary IP addresses per drive: 21nd also appears in the captured packet trace. Similarly,false positive is one that we discover, but does not appear in• Some IP addresses not present in text—on 66 driveshe packet trace. The precision of the validated, highonfidenceaddresses is quite high. Not only are the host’source IP address and the HTTP transfer destination addressesiscovered, we are typically able to discover many of the otherachines in the subnet that are sending broadcasts, forInformation not in the paper:nstance Windows file sharing.• Analysis of next-hop MAC address presence..4. Against real data corpusdigital investigation 8 (2011) S78eS89aving validated our methodology, we apply our networkarving to 1817 images in the Real Data Corpus (Garfinkelt al., August 2009). Note that this corpus contains imagesrom not only general purpose computers, but also fromigital cameras, music players, etc. Specifically, using opertingsystem inference techniques, 335 images are a version ofindows. Of the 1817 images, we discover IP addresses on 723x40%). Those images in which no IP addresses are foundave been excluded from this analysis.Unlike the controlled dataset, we have only limited groundruth over the images in the larger corpus. To ascertain howFig. 4 e Correlation between scanning modalities across723 images in the corpus. Each circle corresponds toa single hard drive, where the X axis indicates the numberof addresses found through binary carving, the Y axisindicates the number of addresses found by ASCII carving,and the size of the circle indicates the number of addressesthat are the same.55

Hibernation decompression found even more packets!Why we focused on hibernation:• Network data structures are in system memory• Memory is stored in hibernation files• Windows overwrites the beginning of hibernation files when resuming—But not the whole file!• Fragments of hibernation files left in unallocated space when windows defragmentsMethodologyHibernation• We find an 8-byte Hibernation XPress Decompression signature within the compressed memory pageheader and decompress the entire page.Opportunistically decompress XPress pagesAddress Count Decompressed Count172.20.105.74 25 600172.20.104.199 41 43418.26.0.230 43 162172.20.20.11 0 4... ... ...Improves recall by an order of magnitude on our test image!56

With cross-drive analysis, we could find drives that hadbeen on the same network.ResultsCross-Drive MAC AnalysisCross-Drive MAC AnalysisMany RDC images bought in batchesWe find 16 Ethernet common between images!Graph shows 8 distinct clusters:IN10-0561.E01cn20-01.aff00:04:ED:66:C7:1900:26:18:BD:D9:E9IN10-0050.E0100:05:5F:EF:14:0100:15:F2:4B:E5:1EIN10-0562.E01IN10-0118.E01IN10-0095.E0100:1E:A6:01:9E:3Ath01-01.affIN10-0010.E01IN10-0048.E01IN10-0014.E0101:00:5E:7F:FF:FAIN10-0052.E0100:1B:B9:9B:D5:BBPS01-021.affil38.affIN10-0413.E01 00:16:76:A2:60:6E IN10-0414.E01il04.aff00:1B:B9:9B:D3:61IN10-0047.E0100:1B:B9:9C:47:67IN10-0049.E01PS01-036.aff00:50:04:EE:6C:F9IN10-0051.E0100:1E:90:D5:EE:5Ecn4-06.aff 00:D0:B7:69:0A:41 1039.affil02.aff00:1E:90:DE:F1:07IN10-0009.E01il42.aff 00:E0:D0:13:14:94 mx5-30.aff00:0E:90:D5:E6:5ER. Beverly, S. Garfinkel, G. Cardwell (NPS) Network Carving DFRWS 2011 25 / 2857

BE 1.3 put MAC addresses in ether.txt andether_histogram.txt files.342699417 00:80:77:31:01:07 n008077310107 1 00:80:77:31:01:07 192.168.1.2 an342700437 00:80:77:31:01:07 the following: 00:80:77:31:01:07 brn008077310107342703371 00:80:77:31:01:07 -s 192.168.1.2 00:80:77:31:01:07 ping 192.168.559251251 00:80:77:31:01:07 N008077310107 1 00:80:77:31:01:07 192.168.1.200:80:77:31:01:07 BRN0080773101684600847 00:80:77:31:01:07 -s 192.168.1.2 00:80:77:31:01:07

BE 1.3 put IP addresses in ip.txt# Filename: charlie-2009-12-11.E01# Feature-Recorder: ip# Feature-File-Version: 1.1117942521 20.137.78.24 struct ip R (src) cksum-bad117942521 94.89.93.194 struct ip L (dst) cksum-bad118342942 20.137.78.24 struct ip R (src) cksum-bad118342942 94.89.93.194 struct ip L (dst) cksum-bad9977306594 192.168.1.1 sockaddr_in9977393926 63.245.209.93 sockaddr_in5839793854-HIBER-17952268 90.4.162.232 struct ip L (dst) cksum-bad5839793854-HIBER-17960460 78.0.3.185 struct ip R (src) cksum-bad5839793854-HIBER-17960460 90.4.162.232 struct ip L (dst) cksum-bad6339825268 192.168.1.104 struct ip L (src) cksum-ok6339825268 192.168.1.1 struct ip R (dst) cksum-ok6339825320 192.168.1.104 struct ip L (src) cksum-ok5839793854-HIBER-129985200• Local (“L”) or Remote (“R”)8.3.2.3 sockaddr_in• chksum-bad/chksum-ok — IP checksum good or bad• sockaddr_in — IP address from sockaddr_in structure.59

ip_histogram.txt removes random noise(1.3 histogram is only of chksum-ok values)Histogram of all values:# Filename: charlie-2009-12-11.E01# Feature-Recorder: ip# Histogram-File-Version: 1.1n=93 108.5.218.9n=93 7.90.102.193n=64 20.137.78.24n=64 94.89.93.194n=31 176.69.248.3n=30 5.225.0.252n=26 120.23.102.15n=26 182.210.102.137n=24 152.6.0.164n=24 152.6.0.220n=19 192.168.1.1n=14 192.168.1.104n=13 141.77.252.81n=13 80.4.139.660

In 2012, scan_net was expanded to create pcap files“pcap” files are much easier to process.• Provides entire packet (ip.txt gave the offset of the packet in the disk image.)• Process with existing tools: Wireshark, tcpflow, tcpdump,...• Include timestamp metadata for each packet (for those from pcap files)inputscan_netTwo sources of packets:• Ethernet or IP packet in memory.• PCAP file on the drive—Someone was running a sniffer?IP addressespcap files61

pcap files are a common file format for packets.There are many tools for analyzing pcap files.$ tcpdump -r packets.pcap-5:-59:-59.0000 IP 192.168.1.1.microsoft-ds > 192.168.1.104.udpradio: Flags [.], ack416616880, win 65535, length 0-5:-59:-59.0000 IP 192.168.1.1.microsoft-ds > 192.168.1.104.udpradio: Flags [.], ack4294967234, win 65535, length 0-5:-59:-59.0000 IP 192.168.1.1.microsoft-ds > 192.168.1.104.udpradio: Flags [.], ack4294967084, win 65535, length 0-5:-59:-59.0000 IP 192.168.1.1.microsoft-ds > 192.168.1.104.udpradio: Flags [P.], seq4294966956:4294967060, ack 4294967008, win 65535, length 104SMB PACKET: SMBtrans2(REPLY)...Notice time is -5:-59:-59.000• This has a time zone of -0600 (I’m typing this in Utah in the summer)• The time in the packet file is “1”/* Possibly a valid ethernet frame but not preceded by a* pcap_record_header.* Write it out with time of 1.*/• Only packets carved from a PCAP file will have a the correct time.62

PCAP carving takes advantage of the headers.PCAP HeaderHeaderPacketThis is a great way to find packet sniffers!HeaderPacketHeaderPacketpcap file format63

We find a lot of packets on disks!August 2013 study of bulk_extractor on corpus.• Total disks processed: 2418• Disks with extractable packets: 710Top 5 drives:DriveIN4001-1026IL2-0086BD1-1071IL3-0212TH0001_0010PCAP size31MB10MB6MB5MB3MB64

Example of packets from IN4001-1026:Packets in file 204,202UDP packets 93,130TCP packets 111,039Sample UDP-5:00:00.000000 IP 10.48.231.2.hsrp >all-routers.mcast.net.hsrp:HSRPv0-hello 20: state=active group=5 addr=10.48.231.1Sample TCP:-5:00:00.000000 IP 10.48.133.228.http > 10.48.231.44.chip-lm:Flags [.], seq 6301:7561, ack 763, win 65535, length 1260Note:• Time -5:00:00, so these packets came from memory, not a carved PCAP file.65

The big picture:using low-probability data for high-outcome correlation.bulk_extractor was created to find low-probably, high-value information.• “low-probability” means does not occur by chance; not part of the background.• Examples:—email addresses—Credit card numbers—Evidence of specific executables (prefetch files; PE headers)Once found, the goal was to enable:• Multi-drive correlation—Find all drives with a specific identifier—“Blind correlation” between multiple drives to find identifiers in commonWe now have more sources of low-probability data:• Fragments from ZIP & RAR files• Ethernet MAC addresses66

In conclusiontcpflow:• Fast offline reassembly of TCP streams• Handles many common situations• Wifi (radiotap & 802.11) support• Easy workflow integration• Does not decrypt SSLTCPflowbulk_extractor 1.4:• Carves ZIP and RAR fragments into files• Carves encrypted RAR files• Carves IP packets and PCAP fragments into PCAP filesContact information: slgarfin@nps.edu ; gdgarsl@dodiis.ic.gov67