Scrapy Is applicable to Python A quick 、 High level screen grabs and web Grabbing framework , Used to grab web Site and extract structured data from the page .Scrapy A wide range of uses , Can be used for data mining 、 Monitoring and automated testing .
We know , It's hard to write a reptile , For example, initiate a request 、 Data analysis 、 Anti reptile mechanism 、 Asynchronous request, etc . If we do it manually every time , It's very troublesome. .scrapy This framework has already encapsulated some basic contents , We can use it directly , Very convenient .
We use the following two figures , Let's get to know each other ;
You can see the above figure ,scrapy It is also composed of many components , Let's take a look at the role of each component ;
Scrapy Engine（ engine ）：Scrapy The core part of the framework . Responsible for Spider and ItemPipeline、Downloader、Scheduler Intermediate communication 、 Transfer data, etc .
Spider（ Reptiles ）： Send the link to be crawled to the engine , Finally, the engine sends the data requested by other modules to the crawler , The crawler parses the data it wants . This part is written by our developers , Because of which links to crawl , Which data in the page we need , It's all up to the programmer .
Scheduler（ Scheduler ）： Responsible for receiving requests from the engine , And arrange and arrange in a certain way , Responsible for scheduling the order of requests, etc .
Downloader（ Downloader ）： Responsible for receiving the download request from the engine , Then go to the network to download the corresponding data and return it to the engine .
Item Pipeline（ The Conduit ）： Responsible for Spider（ Reptiles ） Save the data passed on . Exactly where to keep it , It depends on the developer's own needs .
Downloader Middlewares（ Download Middleware ）： Middleware that can extend the communication between downloader and engine .
Spider Middlewares（Spider middleware ）： Middleware that can extend the communication between engine and crawler .
In this paper, the end , Relevant contents are updated daily .
For more information, go to VX official account “ Operation and maintenance home ” , Get the latest article .
------ “ Operation and maintenance home ” ------
------ “ Operation and maintenance home ” ------
------ “ Operation and maintenance home ” ------
linux Under the system ,mknodlinux,linux Directory write permission , Chinese cabbage can be installed linux Do you ,linux How the system creates files , Led the g linux How to install software in the system ,linux Text positioning ;
ocr distinguish linux,linux Anchoring suffix ,linux System usage records ,u Dish has linux Image file , Fresh students will not Linux,linux kernel 64 position ,linux Self starting management service ;
linux Calculate folder size ,linux What are the equipment names ,linux Can I use a virtual machine ,linux The system cannot enter the command line , How to create kalilinux,linux Follow so Are the documents the same .
