程序師世界是廣大編程愛好者互助、分享、學習的平台,程序師世界有你更精彩!
首頁
編程語言
C語言|JAVA編程
Python編程
網頁編程
ASP編程|PHP編程
JSP編程
數據庫知識
MYSQL數據庫|SqlServer數據庫
Oracle數據庫|DB2數據庫
您现在的位置: 程式師世界 >> 編程語言 >  >> 更多編程語言 >> Python

The scrapy framework only gets a set of results. I dont know why (Language Python)

編輯:Python
The phenomenon and background of the problem
import scrapyclass ThreedmSpider(scrapy.Spider): name = 'threedm' # allowed_domains = ['www.3xxxdm.com'] start_urls = ['https://www.3dmgame.com/bagua_62_1/'] # # Generate a generic URL Templates  # url = "https://www.3dmgame.com/bagua_62_%d/" # page_num = 2 def parse(self, response): li_list = response.xpath("/html/body/div[3]/div[2]/div[2]/ul") for li in li_list: text = li.xpath("./li/a/div[2]/text()").extract_first() print(text) detail_url = li.xpath("./li/a/@href").extract_first() # for detail in detail_url: yield scrapy.Request(url=detail_url, callback=self.parse_detail) # pass # if self.page_num <= 2: # new_url = format(self.url%self.page_num) # self.page_num += 1 # # Manual request sending ,callback Callback functions are specifically used for data parsing  # yield scrapy.Request(url=new_url, callback=self.parse) def parse_detail(self, response): detail = response.xpath("/html/body/div[2]/div[2]/div[3]//text()").extract() detail = ''.join(detail).strip() print(detail)
Operation results and error reporting contents

(venv) PS C:\Users\Administrator\Desktop\douban\threedmPro> scrapy crawl threedm
Stephen Chow 《 God of food 》 Location: Treasure seafood restaurant Officially bid farewell to Hong Kong
Stephen Chow movie 《 God of food 》 Location —— Treasure seafood restaurant , Officially bid farewell to Hong Kong , Transfer to Southeast Asia .

My solution ideas and tried methods
What I want to achieve

  1. 上一篇文章:
  2. 下一篇文章:
Copyright © 程式師世界 All Rights Reserved