爬区国内所有高校名单

python 爬虫 评论

在挖教育src时，发现手动找高校链接太慢，于是网上搜了下在http://www.hao123.com/edu

发现了收录了国内高校的链接即名称，嘿嘿～

爬取各地大学链接

1
2
3
4
5
6
7
8
9
from urllib.request import urlopen#用于获取网页
from bs4 import BeautifulSoup#用于解析网页

html = urlopen('http://www.hao123.com/edu')#爬取带大学url的页面
bsObj = BeautifulSoup(html, 'html.parser')
t1 = bsObj.find_all('a')
for t2 in t1:
    t3 = t2.get('href')
    print(t3)

筛选得到的url

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
http://www.hao123.com/eduhtm/edubj.htm
http://www.hao123.com/eduhtm/edubj02.htm
http://www.hao123.com/eduhtm/edubj03.htm
http://www.hao123.com/eduhtm/edubj04.htm
http://www.hao123.com/eduhtm/edutj.htm
http://www.hao123.com/eduhtm/edutj02.htm
http://www.hao123.com/eduhtm/tj03.htm
http://www.hao123.com/eduhtm/tj04.htm
http://www.hao123.com/eduhtm/hb.htm
http://www.hao123.com/eduhtm/hb02.htm
http://www.hao123.com/eduhtm/hb03.htm
http://www.hao123.com/eduhtm/hb04.htm
http://www.hao123.com/eduhtm/jilin.htm
http://www.hao123.com/eduhtm/jilin02.htm
http://www.hao123.com/eduhtm/jilin03.htm
http://www.hao123.com/eduhtm/jilin04.htm
http://www.hao123.com/eduhtm/numeg.htm
http://www.hao123.com/eduhtm/nmg02.htm
http://www.hao123.com/eduhtm/nmg03.htm
http://www.hao123.com/eduhtm/nmg04.htm
http://www.hao123.com/eduhtm/shnagxi.htm
http://www.hao123.com/eduhtm/sx02.htm
http://www.hao123.com/eduhtm/sx03.htm
http://www.hao123.com/eduhtm/sx04.htm
http://www.hao123.com/eduhtm/edusy.htm
http://www.hao123.com/eduhtm/ln02.htm
http://www.hao123.com/eduhtm/ln03.htm
http://www.hao123.com/eduhtm/ln04.htm
http://www.hao123.com/eduhtm/guansu.htm
http://www.hao123.com/eduhtm/gs02.htm
http://www.hao123.com/eduhtm/gs03.htm
http://www.hao123.com/eduhtm/gs04.htm
http://www.hao123.com/eduhtm/edush.htm
http://www.hao123.com/eduhtm/sh02.htm
http://www.hao123.com/eduhtm/sh03.htm
http://www.hao123.com/eduhtm/sh04.htm
http://www.hao123.com/eduhtm/guangxi.htm
http://www.hao123.com/eduhtm/gx02.htm
http://www.hao123.com/eduhtm/gx03.htm
http://www.hao123.com/eduhtm/gx04.htm
http://www.hao123.com/eduhtm/eduhz.htm
http://www.hao123.com/eduhtm/zj02.htm
http://www.hao123.com/eduhtm/zj03.htm
http://www.hao123.com/eduhtm/zj04.htm
http://www.hao123.com/eduhtm/anfei.htm
http://www.hao123.com/eduhtm/ah02.htm
http://www.hao123.com/eduhtm/ah03.htm
http://www.hao123.com/eduhtm/ah04.htm
http://www.hao123.com/eduhtm/fujian.htm
http://www.hao123.com/eduhtm/fj02.htm
http://www.hao123.com/eduhtm/fj03.htm
http://www.hao123.com/eduhtm/fj04.htm
http://www.hao123.com/eduhtm/jianxi.htm
http://www.hao123.com/eduhtm/jx02.htm
http://www.hao123.com/eduhtm/jx03.htm
http://www.hao123.com/eduhtm/jx04.htm
http://www.hao123.com/eduhtm/xizang.htm
http://www.hao123.com/eduhtm/xz02.htm
http://www.hao123.com/eduhtm/henan.htm
http://www.hao123.com/eduhtm/hn02.htm
http://www.hao123.com/eduhtm/hn03.htm
http://www.hao123.com/eduhtm/hn04.htm
http://www.hao123.com/eduhtm/eduwh.htm
http://www.hao123.com/eduhtm/hubei02.htm
http://www.hao123.com/eduhtm/hubei03.htm
http://www.hao123.com/eduhtm/hubei04.htm
http://www.hao123.com/eduhtm/hunan.htm
http://www.hao123.com/eduhtm/hunan02.htm
http://www.hao123.com/eduhtm/hunan03.htm
http://www.hao123.com/eduhtm/hunan04.htm
http://www.hao123.com/eduhtm/edugz.htm
http://www.hao123.com/eduhtm/guangdong02.htm
http://www.hao123.com/eduhtm/guangdong03.htm
http://www.hao123.com/eduhtm/guangdong04.htm
http://www.hao123.com/eduhtm/xinjang.htm
http://www.hao123.com/eduhtm/xinjiang02.htm
http://www.hao123.com/eduhtm/xinjiang03.htm
http://www.hao123.com/eduhtm/xinjiang04.htm
http://www.hao123.com/eduhtm/educq.htm
http://www.hao123.com/eduhtm/chongqing02.htm
http://www.hao123.com/eduhtm/chongqing03.htm
http://www.hao123.com/eduhtm/chongqing04.htm
http://www.hao123.com/eduhtm/educd.htm
http://www.hao123.com/eduhtm/sichuan02.htm
http://www.hao123.com/eduhtm/sichuan03.htm
http://www.hao123.com/eduhtm/sichuan04.htm
http://www.hao123.com/eduhtm/shangdong.htm
http://www.hao123.com/eduhtm/shandong02.htm
http://www.hao123.com/eduhtm/shandong03.htm
http://www.hao123.com/eduhtm/shandong04.htm
http://www.hao123.com/eduhtm/yunnan.htm
http://www.hao123.com/eduhtm/yunnan02.htm
http://www.hao123.com/eduhtm/yunnan03.htm
http://www.hao123.com/eduhtm/yunnan04.htm
http://www.hao123.com/eduhtm/eduhab.htm
http://www.hao123.com/eduhtm/heilongjiang02.htm
http://www.hao123.com/eduhtm/heilongjiang03.htm
http://www.hao123.com/eduhtm/heilongjiang04.htm
http://www.hao123.com/eduhtm/edunanjin.htm
http://www.hao123.com/eduhtm/jiangsu02.htm
http://www.hao123.com/eduhtm/jiangsu03.htm
http://www.hao123.com/eduhtm/jiangsu04.htm
http://www.hao123.com/eduhtm/eduxia.htm
http://www.hao123.com/eduhtm/shanxi02.htm
http://www.hao123.com/eduhtm/shanxi03.htm
http://www.hao123.com/eduhtm/shanxi04.htm
http://www.hao123.com/eduhtm/gz.htm
http://www.hao123.com/eduhtm/guizhou02.htm
http://www.hao123.com/eduhtm/guizhou03.htm
http://www.hao123.com/eduhtm/guizhou04.htm
http://www.hao123.com/eduhtm/qingdao.htm
http://www.hao123.com/eduhtm/qinghai02.htm
http://www.hao123.com/eduhtm/qinghai03.htm
http://www.hao123.com/eduhtm/qinghai04.htm
http://www.hao123.com/eduhtm/nixai.htm
http://www.hao123.com/eduhtm/ningxia02.htm
http://www.hao123.com/eduhtm/ningxia03.htm
http://www.hao123.com/eduhtm/ningxia04.htm
http://www.hao123.com/eduhtm/hainan.htm
http://www.hao123.com/eduhtm/hainan02.htm
http://www.hao123.com/eduhtm/hainan04.htm

爬取以上中的具体院校信息

1
2
3
4
5
6
7
8
9
10
11
12
import re
import urllib.request

for line in open("1.txt",'rt'):
        res = urllib.request.urlopen(line)
        html = res.read().decode('gb2312')
        rule = re.findall(r"<p> 　　 <a.*?href=.*?<\/a></p></td>", html, re.I|re.S|re.M)
        for value in rule:
                with open('edu-urls.txt', 'a') as f:
                        print(value.strip('<p> 　　 '),file=f)
print("it's ok")
~

得到最后的院校信息edu-urls.txt，一共2865条信息。

整理结果

Download

以上结果中存在百度百科的链接，没有去过滤，如需要请自行过滤。

ShymWeb Penetration & Safe

念念不忘，必有回响。