Download - The Bastards Book of Regular Expressions

Recommendations

Info

Sorting a police blotterThe Colorado Springs Police Department posts six months worth of crime incident reports at itswebsite. The data clearly comes from a database but we only have the text of the webpage to workwith:Colorado Springs Police blotterWe have to essentially reverse-engineer the output on the webpage to recreate the spreadsheet thatthe reports originate from. Unfortunately, copying-and-pasting directly from the webpage does notpreserve any of the structure.However, there are enough patterns in the text glob to do what we need with just regular expressions.The end product will look very, very convoluted. But I’ll show you how it’s actually just a bunch ofsimple pieces all put together.Note: This is just a proof of concept to show you how far you can get with intuitively recognizingpatterns in text. If you are actually trying to make usable data from these pages for a real-life project,this is not how you should do it. Learning a little bit of how to program in order to automatically webcrawland parse the pages – using a scraping library more powerful and easy-to-use than regexes –is the ideal way to do this.But let’s see how far we can get with regexes.153
Sorting a police blotter 154Sloppy copy-and-pasteVisit one of the pages at the Colorado Springs online police blotter²⁸. Each page should have 10reports.If you copy-and-paste the entire webpage into a textfile, though, you get a mess like this (the nameshave been changed from the actual report but the (amazing) events are verbatim):17836 Record ID March 5, 2013 Incident Date 6:59:00 AM Time Sand Creek – ShiftIDivision OtherTitle190X Carmel DriveLocationSand Creek Officers were sent to theWestwind Apartments. The reporting party said her boyfriend was cut with a sword bya neighbor. Medical personnel staged and officers responded to the scene. Investigationrevealed that Sean Doe went to Guy Incognito’s apartment with a Samurai sword. Doeswung the sword at Incognito’s face when Incognito opened his door. Incognito caughtthe blade in both hands and he was able wrestle the sword away. Incognito and Doewere bleeding extensively. Officers arrested Doe at his apartment. Incognito and Doewere transported to Memorial Central for treatment. Both are in stable condition andawaiting surgery.SummarySean DoeAdults Arrested Lt. Strassburg-Aldal - 444-7270PDContact & NumberWhat happened? It’s a consequence of how the HTML is structured, a topic that we don’t need toconcern ourselves with. All that matters is we have a jumble.But luckily, the jumble isn’t too bad. The problem is that the data labels, such as “Record ID” and“Adults Arrested” have been thrown into the text with no separation. However, as long as thosedata labels stay the same for each record, then we can use them to create a usable regex.Below is the jumbled text again, except that I’ve bolded the data labels:17836 Record ID March 5, 2013 Incident Date 6:59:00 AM Time Sand Creek – ShiftIDivision OtherTitle1900 Block of Carmel DriveLocationSand Creek Officers weresent to the Westwind Apartments. The reporting party said her boyfriend was cutwith a sword by a neighbor. Medical personnel staged and officers responded to thescene. Investigation revealed that Sean Doe went to Guy Incognito’s apartment with aSamurai sword. Doe swung the sword at Incognito’s face when Incognito opened hisdoor. Incognito caught the blade in both hands and he was able wrestle the swordaway. Incognito and Doe were bleeding extensively. Officers arrested Doe at hisapartment. Incognito and Doe were transported to Memorial Central for treatment.Both are in stable condition and awaiting surgery.SummarySean DoeAdults ArrestedLt. Strassburg-Aldal - 444-7270PD Contact **& NumberIt appears the data labels come after their associated data, e.g:Time Sand Creek – Shift IDivision²⁸http://www.springsgov.com/units/police/policeblotter.asp
Page 1:
The Bastards Book ofRegular Express
Page 5 and 6:
CONTENTSOptionality and alternation
Page 7 and 8:
CONTENTSSwitching visualizations (T
Page 9 and 10:
Regular Expressions are for Everyon
Page 11 and 12:
Regular Expressions are for Everyon
Page 13 and 14:
Getting Started6
Page 15 and 16:
Finding a proper text editor 8Notep
Page 17 and 18:
Finding a proper text editor 10Edit
Page 19 and 20:
Finding a proper text editor 12Text
Page 21 and 22:
Finding a proper text editor 14does
Page 23 and 24:
Finding a proper text editor 16Yest
Page 25 and 26:
Finding a proper text editor 18A do
Page 27 and 28:
A better Find-and-Replace 20The lim
Page 29 and 30:
A better Find-and-Replace 22Using R
Page 31 and 32:
Your first regex 24helloThe Find-an
Page 33 and 34:
Your first regex 26The regex syntax
Page 35 and 36:
Your first regex 28Double-bounded
Page 37 and 38:
Your first regex 30AnswerFind \bcat
Page 39 and 40:
Removing emptinessIt’s funny how
Page 41 and 42:
Removing emptiness 34Now let’s do
Page 43 and 44:
Removing emptiness 36…as opposed
Page 45 and 46:
Removing emptiness 38Replacement in
Page 47 and 48:
Match one-or-more with the plus sig
Page 49 and 50:
Page 51 and 52:
Page 53 and 54:
Page 55 and 56:
Match zero-or-more with the star si
Page 57 and 58:
Specific and limited repetition 501
Page 59 and 60:
Specific and limited repetition 52E
Page 61 and 62:
Specific and limited repetition 54C
Page 63 and 64:
Anchors: A way to trim emptinessIn
Page 65 and 66:
Anchors: A way to trim emptiness 58
Page 67 and 68:
Page 69 and 70:
Page 71 and 72:
Matching any letter, any number 64A
Page 73 and 74:
Matching any letter, any number 66W
Page 75 and 76:
Matching any letter, any number 68[
Page 77 and 78:
Matching any letter, any number 70R
Page 79 and 80:
Matching any letter, any number 72F
Page 81 and 82:
Matching any letter, any number 74A
Page 83 and 84:
Negative character sets 76$1,200.00
Page 85 and 86:
Negative character sets 78\W+Exerci
Page 87 and 88:
Capture, Reuse 80Find (ba)+Matches
Page 89 and 90:
Capture, Reuse 82.- then the 1st ba
Page 91 and 92:
Capture, Reuse 84In English We are
Page 93 and 94:
Capture, Reuse 86ApplesOraclesOrang
Page 95 and 96:
Capture, Reuse 8805-14-8912-03-9803
Page 97 and 98:
Capture, Reuse 90Mary asked: "What
Page 99 and 100:
Optionality and alternationThe two
Page 101 and 102:
Optionality and alternation 94Answe
Page 103 and 104:
Optionality and alternation 96Studi
Page 105 and 106:
Optionality and alternation 98You m
Page 107 and 108:
Laziness and greediness 100Being to
Page 109 and 110: Laziness and greediness 102Replace
Page 111 and 112: Laziness and greediness 104With the
Page 113 and 114: Lookarounds 106cat(?=s)- will match
Page 115 and 116: Lookarounds 108`(?
Page 117 and 118: Lookarounds 110ExerciseGiven the fo
Page 119 and 120: Lookarounds 112City,CountryAlbuquer
Page 121 and 122: Lookarounds 114Given this list of c
Page 123 and 124: Lookarounds 116100,J.D. Salinger Av
Page 125 and 126: Lookarounds 1186,300|Apples|New Yor
Page 127 and 128: Lookarounds 120^(\d+) Record ID (\w
Page 129 and 130: Lookarounds 122d. Remove all asides
Page 131 and 132: From Data to HTML (TODO)This chapte
Page 133 and 134: From Data to HTML (TODO) 126Turning
Page 135 and 136: The ExercisesI’ve never been one
Page 137 and 138: Data Cleaning with the Stars 130LOR
Page 139 and 140: Data Cleaning with the Stars 132[GM
Page 141 and 142: Finding needles in haystacks (TODO)
Page 143 and 144: Changing phone format (TODO)Todo: T
Page 145 and 146: Changing phone format (TODO) 138Exe
Page 147 and 148: Changing phone format (TODO) 140Ans
Page 149 and 150: Changing phone format (TODO) 142`1-
Page 151 and 152: Changing phone format (TODO) 144(\d
Page 153 and 154: Dating, Associated Press Style (TOD
Page 159: Dating, Associated Press Style (TOD
Page 163 and 164: Sorting a police blotter 156I used
Page 165 and 166: Converting XML to tab-delimited dat
Page 167 and 168: Converting XML to tab-delimited dat
Page 169 and 170: Cleaning up Microsoft Word HTML(TOD
Page 171 and 172: Cleaning up OCR Text (TODO)Image sc
Page 173 and 174: Moving forwardThank you for taking
show all

Download - The Bastards Book of Regular Expressions

Create successful ePaper yourself

Delete template?

Save as template?