The goal is ： Get Taobao Search page information , Extract the commodity name and price
understand ： Taobao's search interface
Writing regular expressions
Direct crawling cookie Value is overdue , The output is empty , So we need to header
solve ： In the desired crawl page , Press F12, Click on network, Refresh the page , Click on the search?q=, Turn down to see cookie And user-agent
Find the page spacing value 
# =============================================
# --*-- coding: utf-8 --*--
# @Time : 2021-02-14
# @Author : Oraer_
# @CSDN : https://blog.csdn.net/Oraer
# @FileName: Taobao commodity price comparison directional crawler .py
# @Software: PyCharm
# =============================================
import re
import requests
def getHTMLText(url, header):
try:
r = requests.get(url, headers=header, timeout=30)
r.raise_for_status()
r.encoding = r.apparent_encoding
return r.text
except:
return ""
def parsePage(ilt, html):
try:
plt = re.findall(r'\"view_price\"\:\"[\d\.]*\"', html) # Price
tlt = re.findall(r'\"raw_title\"\:\".*?\"', html) # Name of commodity
for i in range(len(plt)):
price = eval(plt[i].split(':')[1]) # Separate prices from strings
title = eval(tlt[i].split(':')[1]) # eval()
ilt.append([price, title]) # Put each pair of prices and product names Add to the list ilt
except:
print("...")
def printGoodsList(ilt):
tplt = "{:4}\t{:8}\t{:16}" # Header
print(tplt.format(" Serial number ", " Price ", " Name of commodity "))
count = 0
for g in ilt:
count = count + 1
print(tplt.format(count, g[0], g[1]))
def main():
goods = ' Huawei ' # Crawling for goods
depth = 2 # Depth of climb
start_url = 'https://s.taobao.com/search?q=' + goods
header = {
"Cookie": 'thw=cn; cna=l4qWGUADnlsCAdpNONov0COS; sgcookie=E100amQ%2FAqzHrAWNxy6h4p2Pjb8E4ps5AMe5dY7LTxvY5u78KCAW52HbJyHBByU7qNOtGW%2BIgHtGXv6vpAL71HifU1yFtej6bQezPELhj7XDCOU%3D; tracknick=tb956885452; _cc_=UtASsssmfA%3D%3D; enc=IgiMy3EBqFA9uaISLYfat5cWOuCoWLTHbC8TLE9Rn5v5Ev%2Bgjsi10obFxjMaKoCYmDwApXjgWF4ni9XWrwe%2FNXGKL69AShbXmuWlLvXFM80%3D; hng=CN%7Czh-CN%7CCNY%7C156; miid=3845148332138405182; _m_h5_tk=8f22d821383d7a5ea68960d13c500f93_1644847346926; _m_h5_tk_enc=497b6b19efc8a329b36db37d929954ca; xlly_s=1; alitrackid=www.taobao.com; lastalitrackid=www.taobao.com; cookie2=1f09b4a01bff2126f0a64239b78b39ba; t=c81afa8b8c905660e9138e49e7735d8b; _tb_token_=faae5d4ef43f4; JSESSIONID=90DC4AD7D51644F3EEAC9AB890CC322F; tfstk=ceJdBQbK7V0nOOs90BhMNv9s3LJdZIZdrk_8ecovmfY7JayRi8vDHrKoOg7VJcC..; l=eBMZ8HgggLfBS9DBBOfZlurza77O9IRbouPzaNbMiOCPOz5e5pnNW6caISYwCnGVHsK2R3uqBzNwBf8GlyCq0-Y3L3k_J_VsndC..; isg=BGBg3Ct1FqnH86moFMsBh4k3MW4yaUQzqWf6HtpxEXsO1QH_gnphw2GnaX3V4_wL',
"User-Agent": 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.140 Safari/537.36 Edge/18.17763'
} # send out Headers Request And need cookie Information
infoList = [] # Commodity name and price
for i in range(depth):
try:
url = start_url + '&s=' + str(44*i) # Connect to crawl pages Page spacing 44
html = getHTMLText(url, header) # Get page text information
parsePage(infoList, html) # Name the product Price Extract to infoList
except:
continue
printGoodsList(infoList) # Print crawling results
if __name__ == '__main__':
main()
Serial number Price Name of commodity
1 6699.00 Huawei MateBook14s 512G Sharp torch graphics card 14.2 Inch Touch screen intelligent performance lightweight book
2 17799.00 Huawei folding screen mobile phone Mate X2 Flip phone Huawei x2 Huawei 5G mobile phone
3 888.00 Huawei/ Huawei Enjoy 20 Pro Official authentic flagship mobile phone
4 2379.00 SF issued on the same day + Reducible 400 HONOR/ glory 60 5G The official website of the official flagship store of the mobile phone, the new authentic smart surface of all Netcom 50pro The glory of non Huawei mobile phones 60pro
5 2799.00 24 Interest free period Send broken screen treasure Huawei/ Huawei nova 9 mobile phone Huawei mobile phones Huawei nova9 Huawei nova9 mobile phone Huawei nova9pro Huawei's official flagship store
6 4499.00 Huawei HUAWEI MatePad Pro 12.6 Inch Hongmeng HarmonyOS System OLED Screen tablet learning 8GB12GB Memory
7 11588.00 On the same day 【 Give me a broken screensaver 24 Interest free period 】 Huawei /HUAWEI Mate 40 RS 5G Official flagship store of mobile phone genuine Porsche new product Pro Official website collection version
8 19088.00 24 Interest free period 【 On the same day 】HUAWEI/ Huawei Mate X2 5G The new flagship store on the official website of the official authentic mobile phone mate30e Folding screen matex2 The collection edition PU Phloem
9 2099.00 HUAWEI/ Huawei nova 8 SE Huawei mobile phone large screen 4G a phone fit all kinds of networks 6400 Wan HD four shot 66W Super quick charge OLED Huawei's official flagship store
10 5188.00 24 Interest free payment in installments 0 A down payment HUAWEI/ Huawei Mate 40E 5G The official flagship store of smart phones is the authentic Kirin flagship chip curved screen mate40e
11 2099.00 Send broken screen treasure 【 Discount 200 element 】Huawei/ Huawei Nova8 SE The official flagship store official website authentic new all Netcom mobile phone drops directly 66W Quick recharge student game big screen Huawei
12 1299.00 vivo WATCH 2 New independent communication super long endurance intelligent sports Bluetooth health watch official flagship genuine waterproof compatible with Huawei
13 2699.00 Reducible 150 element Spend bai 24 Interest free period HONOR/ glory 60 Pro mobile phone 5G The official website of the official flagship store is the same as that of Gong Jun 50pro Not Huawei's glory 60pro
14 10088.00 On the same day 【24 Interest free period / Reducible 1200】HUAWEI/ Huawei P50 Pocket The official flagship store of treasure box folding mobile phone is authentic p50pro Hongmeng new folding screen customized version
15 1379.00 Reducible 160 element + SF issued on the same day honor/ glory X30i 5G Mobile phone glory official flagship store official website new product launch authentic glory x20 Intelligence is not Huawei's new glory x30i
16 2799.00 On the same day 【24 Interest free gift of broken screen treasure 】Huawei/ Huawei nova 9 The official flagship store of mobile phones is authentic nova9pro Curved panel 7 New model on Hongmeng official website p50 System nova8
17 2099.00 Huawei/ Huawei nova 8 SE4G Mobile phone all Netcom authentic student photo game mobile phone nova8se
18 2499.00 Huawei/ Huawei MatePad 11 The tablet 10.95 Inch matepad Hongmeng office study
19 1488.00 Huawei/ Huawei watch WATCH GT3/GT Collection version of sports intelligence gt3 Blood oxygen monitoring Bluetooth call
20 6188.00 Huawei/ Huawei P50 mobile phone 4G The official flagship store of all Netcom carries Hongmeng p50 A smart phone
21 3499.00 Huawei/ Huawei nova9pro 4G Mobile phone Xiaolong Hongmeng HarmonyOS New product intelligence Nova9Pro
22 2999.00 【 Exclusive for mobile users 】HONOR/ glory 60 5G The official website of the official flagship store of the mobile phone, the new authentic smart surface of all Netcom 50pro The glory of non Huawei mobile phones 60pro
23 6199.00 【24 Interest free period Collect securities 88 Securities Province 100 element 】Huawei/ Huawei Mate 40 pro 5G Genuine mobile phones mate40pro New product mate40pro The mobile phone goes straight down mate40
24 2599.00 【24 Interest free orders minus 150 】 Huawei tablet MatePad11 Tablet computer two in one 10.95 Inch 2021 The new pad Network wide call pro Student board game Office
25 4649.00 【M40pro+ Black flash 】Huawei/ Huawei Mate 40 Pro+ MATE40 pro mate40
26 3799.00 【 Consultation minus 600】 Huawei MatePadPro The tablet 10.8 Inch 2021 The new pad Two in one office student ipad pro Official flagship store authentic matepad11
27 3799.00 【 Consultation minus 600】 Huawei tablet MatePadPro 10.8 " 2021 The new pad Tablet computer two in one Pro Student office iPad Official flagship authentic matepad11
28 2899.00 Huawei Matebook14/13 i7 Light and glorious dragon touch screen i5 New laptop for student Games
29 2780.00 Huawei HUAWEI Huawei Mate30Pro 5G Official authentic flagship mobile phone mate30 Original packaging
30 3288.00 Interest free by installments Huawei/ Huawei p40 pro 5G New official flagship luxury gift P40pro+ mobile phone
31 5999.00 New product Huawei/ Huawei P50 Pro The official website of the official flagship store of mobile phones is authentic Huawei P50 Hongmeng system
32 2889.00 Huawei Matebook13/14s New product i5/7 Ruilong version of student office ultra-thin full screen laptop
33 2599.00 【 Ask for a large discount ！】 Huawei watch Watch 3 pro intelligence eSIM Independent call, heart rate, blood glucose and oxygen detection, official flagship of men's and women's sports bracelet, Hongmeng
34 1899.00 Delivery on the same day 【24 installment 】HONOR/ glory 50 Mobile phone series v50 The new 5G Ten straight drops pro Quality goods SE Curved screen 40 The official website of Huawei's flagship mobile phone store is all Netcom
35 228.00 【2022 The new 】 Huaqiangbei s7 A watch can answer and make calls DT7MAX intelligence PD7MAX+ Bluetooth to measure heart rate and blood pressure DT7proMAX NFC Applicable to apple and Huawei
36 2599.00 Huawei watch Watch 3 pro intelligence esim Independent telephone conversation men and women Bluetooth SPORTS BRACELET heart rate and blood oxygen detection official flagship genuine spot Hongmeng [966]
37 1988.00 HUAWEI Huawei Mate 30 Pro 5G kirin 990 Curved panel E Leica 40 The camera 5G mobile phone
38 6499.00 Huawei/ Huawei MateBook 14 Eleven generations I5 Students' thin office business notebook 2021 paragraph
39 5258.00 New product Huawei/ Huawei Mate 40 5G mobile phone mate40PRO+ Porsche design mate40e
40 1299.00 【 SF issued on the same day + Send glory headphones 】HONOR/ glory X20 5G The official website of Huawei mobile official flagship store x20se Old people and students can play with smart phones for a thousand yuan 20se
41 2628.00 HUAWEI Mate30 mate30pro 5G All Netcom straight screen, curved screen and waterproof Huawei official genuine mobile phones
42 898.00 【 SF issued on the same day + Send a broken screensaver 】HUAWEI/ Huawei Changxiang 20 SE Enjoy the official website of the official flagship genuine mobile store 20se Directly drop the new Huawei student game phone
43 1528.00 Huawei/ Huawei nova 7 5G All Netcom Unicorn 985 Official genuine curved screen nova7pro mobile phone
44 917.00 honor glory glory 9x Genuine lifting camera 4G Smart student Huawei mobile phone package
45 699.00 Huawei HUAWEI Enjoy 9 PLUS Imagination 9S Quality goods 10s The official website of the official flagship store of elderly smart phones
46 3199.00 Spot express Huawei/ Huawei p40 pro 5G mobile phone P40 Official flagship full screen P40 The new
47 1899.00 Official website honor/ glory glory 60 Pro mobile phone 5G Huawei glory, the official flagship store 60 Xiao dragon 778G+
48 2504.00 【 More favorable consultation 】 Huawei tablet MatePad11 New tablet 10.95 Inch HUAWEI Hongmeng studies office painting 2 Online course Pro Students have a comprehensive screen ipad official
49 2504.00 【 More favorable consultation 】 Huawei tablet MatePad11 New tablet 10.95 Inch HUAWEI Hongmeng studies office painting 2 Online course Pro Students have a comprehensive screen ipad official
50 2599.00 The new Huawei/ Huawei NOVA 9 The official website of the official flagship store of mobile phones is authentic Hongmeng Hi Nova9Pro
51 13088.00 New Huawei Mate X2 Collection folding screen Huawei/ Huawei Mate X2 generation 5G kirin 9000 The flagship
52 8999.00 New product Huawei/ Huawei P50 pocket Folding screen kylin 9000 Huawei p50 Treasure box folding mobile phone
53 1388.00 【 The astronaut dial !】 Huawei watch Watch GT2 Sports smart phone watch 3pro Bluetooth call LCD business men's and women's bracelet blood oxygen official authentic flagship spot 46mm
54 3199.00 Huawei/ Huawei p40 pro The official flagship mobile phone of Hongmeng system 5G All Netcom Huawei p40 The new
55 2383.00 The spot Huawei/ Huawei nova 9 Official authentic flagship store new products nova9 Pro Xiaolong mobile phone
56 1999.00 【 Gift giving 】 Huawei tablet MatePad10.4 Inch tablet 2021 Hongmeng two in one for students to learn 11 Official flagship 10 New all Netcom pro Full screen ipad
57 1899.00 【 Spot express 】 Huawei tablet MatePad The tablet 10.4 Inch 2021 New Hongmeng OS education ipad11 Two in one mobile phone calls students all over the network Pro Official flagship
58 5988.00 24 Interest free period 【 The broken screen shall be shipped on the same day 】Huawei/ Huawei P50Pro All mobile phone Netcom Huawei p50 Treasure box Huawei p50 pro The official website of Huawei's official flagship store
59 7738.00 【 From stock 】Huawei/ Huawei Mate 40 Pro 5G The official flagship store of mobile phones mate40pro Official website P50 Quality goods 40e Huawei mate40
60 5499.00 Huawei /HUAWEI MateBook 13 11 Generation core 8GB/16GB+512GB Sharp torch graphics card notebook computer 2K Touch eye protection lightweight office full screen wifi 6
61 6499.00 Huawei laptop HUAWEI MateBook 13s The first 11 generation intel Core processor 16GB+512GB Sharp torch graphics card 13.4 Inch 2.5K Performance thin book
62 2699.00 SF express sends new products of shatenbao to the market Huawei/ Huawei nova 9 The official website of the official flagship store of mobile phones is authentic and new nova9pro Series direct drop 120Hz Hongmeng nova8
63 2699.00 On the same day 【24 Interest free period 】Huawei/ Huawei nova 9 The new mobile phone is official nova8pro Flagship store nova8se series 5G Hongmeng p Official website down 40 Quality goods
64 2799.00 SF Express 5G New products are sent to smash screen treasure Huawei/ Huawei nova 8 Pro The official flagship store of mobile phones has a new version of genuine curved screen nove8 Straight down 9 Official website nova8 series pro
65 2469.00 【 Reducible 350 element + Spend bai 24 installment 】HONOR/ glory 60 5G The official website of the official flagship store of mobile phones is authentic and new 50 Series curved screen is not Huawei's glory 60pro
66 6688.00 It can be reduced on the same day 720 element [24 Interest free period ]Huawei/ Huawei P50 Pro Official flagship website of mobile phone p40pro The new 5g Straight down Collection Edition treasure box pocket Quality goods
67 1319.00 Reducible 70 element + SF issued on the same day HONOR/ glory X30 5G The official website of the official flagship store of mobile phones is authentic and new x30i Smart students are not Huawei's glory x30 mobile phone max 30i
68 1488.00 On the same day 【24 Give gifts in installments 】 Huawei watch Watch GT3 Smart phone watch monitors the heart rate and blood oxygen of men and women's sports Bluetooth bracelet 2p Official flagship authentic 46mm
69 5088.00 On the same day 【24 Interest free period can be reduced 750】Huawei/ Huawei Mate 40 pro 5G The official flagship store of mobile phones is authentic mate50 Hongmeng 40e Official website down p50 60 M40
70 6099.00 HUAWEI/ Huawei Mate 40 Pro 4G Mobile intelligent system mate40pro 4g Curved screen Hongmeng
71 2599.00 Huawei watch Watch3/Pro intelligence esim Independent call mobile Bluetooth for men and women watch3/pro
72 999.00 Huawei/ Huawei Changxiang 20E mobile phone 4G New products of all Netcom 20e official 4g Authentic old people and students imagine
73 10988.00 Huawei/ Huawei P50 Pocket New treasure box folding screen mobile phone Hongmeng system official self operated store p50
74 2988.00 Huawei blood pressure watch WATCH D Sports smart wrist ECG recorder flagship Bracelet official watch d
75 6399.00 Official genuine Huawei/ Huawei Mate40 pro+ a phone fit all kinds of networks 5G mobile phone mate40 Populus euphratica in autumn and summer
76 3738.00 Official original Huawei/ Huawei P40 The flagship 5GP40pro kirin 990 All Netcom mobile phones are in stock
77 2499.00 Huawei MatePad 11 The tablet 10.95 Inch 2021 The new HarmonyOS Hongmeng 120Hz High refresh rate pad Learn the official flagship store of two in one office games ipad
78 1569.00 Huawei HUAWEI HUAWEI P30 PRO Mobile phone curved screen P30 Official genuine Hongmeng mobile phone P40
79 1388.00 Huawei/ Huawei Mate 20 Pro Official genuine smart phone Huawei mobile phone Mate20
80 2699.00 Huawei laptop Matebook13/D14 Light and convenient for students to work by hand 2022 New glory
81 9299.00 Spot SF parcel post Huawei/ Huawei P50 Pocket Folding screen mobile phone P50 Treasure box P50pocket
82 369.00 【 Official genuine 】 Huawei mobile phone universal student phone watch for junior and senior high school students 4G All Netcom video call children's smart watch waterproof positioning multi-function
83 12688.00 mate X2 The collection edition Huawei/ Huawei Mate X2 kirin 9000 Official flagship 5G Second generation folding screen
84 2988.00 Huawei watch WATCHD Huawei wrist ECG and blood pressure recorder watch d Huawei watches are smart
85 2318.00 Huawei/ Huawei Huawei MatePad11 Tablet computer Hongmeng system 2021 Special for students' study
86 5509.00 【 Normal delivery 】Huawei/ Huawei MateBook E 2022 paragraph Laptop tablet 2 close 1
87 4448.00 The spot Huawei/ Huawei P50 Pro Official authentic mobile phones P50Pocket Folding screen 4G kirin 9000
88 1278.00 The spot honor/ glory glory X30 5G Mobile phone honors the official flagship store genuine smart Huawei X30i
89 2099.00 Interest free by installments / Send luxury gifts ! Huawei/Nova8Pro 8+256G 6.72 Curved screen fast charging 5g It is the official authentic product of the flagship mobile phone store 50pro 9
90 1299.00 On the same day ! Send luxury gifts Huawei/ Huawei Changxiang 20Pro 8+128G 6.5 Inch 4800W Pixels 5g The official website of Huawei's official flagship mobile phone store is authentic and excellent
91 1899.00 【 Consulting gifts 】 Huawei tablet MatePad The tablet 10.4 Inch 2021 New Hongmeng education 11 Learn to fully screen all Netcom Huawei mobile phone ipad Two - Pro
92 855.00 SF Express + Risk of screen breakage HONOR/ Glory chang play 20 The flagship store on the official mobile phone website enjoys 20 The new student old man thousand yuan machine is honored 20Pro Smart non Huawei mobile phones
Process ended , The exit code is 0
