Measuring the Goals and Incentives of Local Chinese Officials

import BeautifulSoup
import re
import chardet
’Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.1)
Gecko/2008071615 Fedora/3.0.1-1.fc9 Firefox/3.0.1’)]
Newlines = re.compile(r’[\r\n]\s+’)
def getPageText(url):
data = br.open(url).read()
dataD = data.decode(chardet.detect(data)[’encoding’], ’ignore’)
bs = BeautifulSoup.BeautifulSoup(dataD,
convertEntities=BeautifulSoup.BeautifulSoup.HTML_ENTITIES,
fromEncoding=chardet.detect(data)[’encoding’])
for s in bs.findAll(’script’):
s.replaceWith(’’)
txt = bs.find(’body’).getText(’|’)
txtD = txt.encode(’utf-8’)
return Newlines.sub(’|’, txtD)
link = raw_input()
import time
import random
sec = random.uniform(1,5000)
time.sleep(sec)
text = {}
try:
text[link] = getPageText(link)
except:
text[link] = "||||ERROR||||" + str(sys.exc_info()[0])
C
Data Cleaning Methods
The raw content of these web pages was prepared for text analysis by removing non-
Chinese characters, segmenting the Chinese characters to identify words, and removing
stopwords.
All non-Chinese characters and alphanumeric characters were removed from the raw
output of the web scraping to improve the ease of parsing and tokenization. Content
in languages such as Tibetan, Uyghur, a Turkic language used in Xinjiang, and Korean,
which is used in Jilin province near the border with Korea, were removed. 12
Chinese words can be composed of single or multiple characters, but there are no
12 Existing methods of multilingual processing remain limited, especially for languages that differ as
much as Chinese, Korean, Tibetan, and Uyghur.
23