A Library for Processing

A Library for Processing
Ad-hoc Data in Haskell
Embedding a Data Description Language
Yan Wang and Veronica Gaspes
Halmstad University, Sweden
IFL2008, Sep 12 2008
Yan Wang, Halmstad University, IFL08

Data Is Everywhere
• Standardized data formats
– HTML
– JPEG&MPEG
– XML
– Databases
– … …
• Tools
– Visualizers
– Languages
– Standard libraries
– Transformers
– … …
...
xml
databases
Yan Wang, Halmstad University, IFL08

Data Is Everywhere
• Ad-hoc data formats (Non-standard data formats)
– In geography
– In chemistry
– In genetics
– In finance
– … …
• Tools not available
– Parsers
– Queriers
– Visualizers
– Transformers
– … …
Yan Wang, Halmstad University, IFL08

Ad-hoc Data in Business
Train e-ticket
Flight e-ticket
Time
Date
Departure&Arrival
Transport
Yan Wang, Halmstad University, IFL08

Ad-hoc Binary Data in Networks
YMSG Packet -- Yahoo Instance Message
Src 83.178.165.157
Dst 76.13.15.53
Yan Wang, Halmstad University, IFL08

Existing Approaches
• Conventional languages
– C, Java, etc.
– Time consuming & error-prone
• Traditional Parsers
– Yacc, Happy, Parsec
– Heavy-weight
• Data Decription Languages
– PADS, Datascript, Packettype
– Difficult to extend
Yan Wang, Halmstad University, IFL08

Data Description Calculus
• DDC: the calculus of dependent types for describing data.
– Base types: atomic pieces of data, e.g., intFW(3), stringUtil(ʻ.ʼ)
– Type constructors: richer structures, e.g., {x:intFW(3)| x

Our Approach
• Embedding a DDL into Haskell
– Data formats are described in dependent types using
• Primitive parsers (Base types)
• Parser combinators (Type constructors)
Type Description
Parser
[t] AdhocParser t a
τ
Representation
a
PD
Parse Descriptor
Yan Wang, Halmstad University, IFL08

• C(e)
•Examples
123456
Base Types
class Basetype t a where ... ...
base :: (Basetype t a) =>AdhocParser t a -- [C()]
baselen :: (Basetype t a) => Int AdhocParser t a -- [C(n)]
baseend :: (Basetype t a) => t AdhocParser t a -- [C(t)]
string(n)
stringlen :: Int AdhocParser Char String
stringlen = baselen
stringlen 5
654,321
int(t)
intlen :: Char AdhocParser Char Int
intlen = baseend
intend ’,’
Yan Wang, Halmstad University, IFL08

• {x :τ| e}
• Examples
Constraint
constrainp :: (a Bool) -- e
AdhocParser t b -- [τ]
AdhocParser t (Either a a) -- [{x :| e}]
123,456
654,321
{x :int() | \x x>0 && x0 && x

Dependent Pairs
• Σ x :τ1 .τ2
sigmap :: AdhocParser t a -- [τ1]
(a AdhocParser t b) -- [τ2 (x)]
AdhocParser t (a, b) -- [Σ x :1 .2]
• Examples
’HELLO’
”HELLO”
5HELLO
6HELLO.
Σ x :char().stringend(x)
s1 = sigmap char stringend
Σ len :int().stringlen(len)
s2 = sigmap int stringlen
Yan Wang, Halmstad University, IFL08

Union
• τ1 +τ2
orp :: AdhocParser t a -- [τ1]
AdhocParser t a -- [τ2]
AdhocParser t (Either a b) -- [τ1 + τ2]
• Examples
123.45 is a float
100 is not a float.
float() + int()
num = orp float int
Yan Wang, Halmstad University, IFL08

Adding Tools
Ad-hoc
Data
[t]
Parser
Type Description
τ
AdhocParser t a
Pretty
Printer
Representation
a
Parse Descriptor
Pretty Document
Doc
Error Report
PD
ErrRep
Error
Reporter
Yan Wang, Halmstad University, IFL08

Example: YMSG Packet
40 fe 20 00 06 00 00 00
06 00 00 00 08 00 45 00
00 83 c7 f6 40 00 80 06
e8 ec 53 b2 9a 9d 4c 0d
0f 35 0d 25 13 ba a0 d4
11 3f c7 5d 20 be 50 18
f7 29 dd 17 00 00 59 4d
53 47 00 0f 00 00 00 47
00 06 5a 55 aa 56 00 49
c6 af 31 c0 80 77 61 6e
67 6b 69 74 38 36 c0 80
35 c0 80 74 61 72 65 6b
31 32 61 6c 79 c0 80 31
34 c0 80 48 65 6c 6c 6f
c0 80 39 37 c0 80 31 c0
80 36 33 c0 80 3b 30 c0
80 36 34 c0 80 30 c0 80
32 30 36 c0 80 31 c0 80
YMSG
TCP
IP
Yan Wang, Halmstad University, IFL08

Example: YMSG Packet
40 fe 20 00 06 00 00 00
06 00 00 00 08 00 45 00
00 83 c7 f6 40 00 80 06
e8 ec 53 b2 9a 9d 4c 0d
0f 35 0d 25 13 ba a0 d4
11 3f c7 5d 20 be 50 18
f7 29 dd 17 00 00 59 4d
53 47 00 0f 00 00 00 47
00 06 5a 55 aa 56 00 49
c6 af 31 c0 80 77 61 6e
67 6b 69 74 38 36 c0 80
35 c0 80 74 61 72 65 6b
31 32 61 6c 79 c0 80 31
34 c0 80 48 65 6c 6c 6f
c0 80 39 37 c0 80 31 c0
80 36 33 c0 80 3b 30 c0
80 36 34 c0 80 30 c0 80
32 30 36 c0 80 31 c0 80
1
type HexChar = Char
instance Basetype HexChar Int where
... ...
instance Basetype HexChar Char where
... ...
Yan Wang, Halmstad University, IFL08

Example: YMSG Packet
40 fe 20 00 06 00 00 00
06 00 00 00 08 00 45 00
00 83 c7 f6 40 00 80 06
e8 ec 53 b2 9a 9d 4c 0d
0f 35 0d 25 13 ba a0 d4
11 3f c7 5d 20 be 50 18
f7 29 dd 17 00 00 59 4d
53 47 00 0f 00 00 00 47
00 06 5a 55 aa 56 00 49
c6 af 31 c0 80 77 61 6e
67 6b 69 74 38 36 c0 80
35 c0 80 74 61 72 65 6b
31 32 61 6c 79 c0 80 31
34 c0 80 48 65 6c 6c 6f
c0 80 39 37 c0 80 31 c0
80 36 33 c0 80 3b 30 c0
80 36 34 c0 80 30 c0 80
32 30 36 c0 80 31 c0 80
2
intlen :: Int -> AdhocParser HChar Int
intlen = baselen
charlen :: Int -> AdhocParser HChar Char
charlen = baselen
… …
Yan Wang, Halmstad University, IFL08

Example: YMSG Packet
40 fe 20 00 06 00 00 00
06 00 00 00 08 00 45 00
00 83 c7 f6 40 00 80 06
e8 ec 53 b2 9a 9d 4c 0d
0f 35 0d 25 13 ba a0 d4
11 3f c7 5d 20 be 50 18
f7 29 dd 17 00 00 59 4d
53 47 00 0f 00 00 00 47
00 06 5a 55 aa 56 00 49
c6 af 31 c0 80 77 61 6e
67 6b 69 74 38 36 c0 80
35 c0 80 74 61 72 65 6b
31 32 61 6c 79 c0 80 31
34 c0 80 48 65 6c 6c 6f
c0 80 39 37 c0 80 31 c0
80 36 33 c0 80 3b 30 c0
80 36 34 c0 80 30 c0 80
32 30 36 c0 80 31 c0 80
3.1
ippacket =
do version constrainp (==4) (intlen 1)
ihl intlen 1
... ...
tlen intlen 4
src seqp unit unit (intlen 2)
(\xs -> length xs == 4)
dest … …
options orp
(seqp unit unit (intlen 8)
(\xs length xs == (ihl-5)))
unit
(port, sender, reciever, msg) tcppacket
return (Ymsg src dest sender reciever msg)
YMSG
TCP
IP
Yan Wang, Halmstad University, IFL08

Example: YMSG Packet
40 fe 20 00 06 00 00 00
06 00 00 00 08 00 45 00
00 83 c7 f6 40 00 80 06
e8 ec 53 b2 9a 9d 4c 0d
0f 35 0d 25 13 ba a0 d4
11 3f c7 5d 20 be 50 18
f7 29 dd 17 00 00 59 4d
53 47 00 0f 00 00 00 47
00 06 5a 55 aa 56 00 49
c6 af 31 c0 80 77 61 6e
67 6b 69 74 38 36 c0 80
35 c0 80 74 61 72 65 6b
31 32 61 6c 79 c0 80 31
34 c0 80 48 65 6c 6c 6f
c0 80 39 37 c0 80 31 c0
80 36 33 c0 80 3b 30 c0
80 36 34 c0 80 30 c0 80
32 30 36 c0 80 31 c0 80
3.2
tcppacket =
do
... ...
port constrainp (== 5050) (intlen 1)
... ...
(sender, reciever, msg) ymsgpacket
return (port, sender, reciever, msg)
ymsgpacket =
do
... ...
return (sender, reciever, msg)
YMSG
TCP
IP
Yan Wang, Halmstad University, IFL08

Example: YMSG Packet
40 fe 20 00 06 00 00 00
06 00 00 00 08 00 45 00
00 83 c7 f6 40 00 80 06
e8 ec 53 b2 9a 9d 4c 0d
0f 35 0d 25 13 ba a0 d4
11 3f c7 5d 20 be 50 18
f7 29 dd 17 00 00 59 4d
53 47 00 0f 00 00 00 47
00 06 5a 55 aa 56 00 49
c6 af 31 c0 80 77 61 6e
67 6b 69 74 38 36 c0 80
35 c0 80 74 61 72 65 6b
31 32 61 6c 79 c0 80 31
34 c0 80 48 65 6c 6c 6f
c0 80 39 37 c0 80 31 c0
80 36 33 c0 80 3b 30 c0
80 36 34 c0 80 30 c0 80
32 30 36 c0 80 31 c0 80
ippacket
Yahoo msg in IPv4:
from Alice (83.178.165.157)
to Bob (76.13.15.53)
on port 5050
msg is ”Hello”
Yan Wang, Halmstad University, IFL08

Thanks for your attention!
Questions & Suggestions?
Yan Wang, Halmstad University, IFL08

Implementation
• newtype AdhocParser t a
= P (([t], PD) -> (Either String a, [t],PD)))
• newtype PD = MkPD Int ErrCode Span Body
newtype ErrCode = Ok | Err | Fail
type Span = (Offset, Offset)
data Body = Unit | Pair PD PD | Or (Either PD PD)
| Constrain PD | Seq Int [PD]
| Scan (Maybe (Int,PD)) | Struct [PD]