Crawling https The trouble must be more than http Only a lot more , Then why don't we put https Switch to http？
Crawl some https When linking, it will report requests.exceptions.SSLError Error of , But it can be solved without verification （verify=False), that https Than http Is it safe? ？
If you climb Baidu or csdn You will find that the link uses http and https It doesn't make any difference , The result of crawling is the same . In this case , All the websites we crawl put https Switch to http Beauty is not true ？ I will not report SSLError Errors don't warn , But it's obviously impossible . Because it is the website that decides you to use http Agreement or https agreement , If the site does not allow you to use http agreement , When climbing url still http://xxxxx.xxx Then you won't get the page you want .
Actually verify=False The function is equivalent to browsing a https After the web browser intercepts the warning to you , You clicked continue to . But it doesn't mean that the certificate of his website is a decoration , The encrypted transmission will still encrypt the transmission . So why does the browser warn that the site is unsafe ？ Because the certificate received by the browser is not issued by an institution in the trusted institution list , He may be a certificate issued by the wildlife agency , It may also be a self-made certificate , It could also be a hacker's certificate . So the browser warning is not safe mainly because it is afraid that hackers instead of web servers may give you certificates .
It's a bit messy , There were many questions at that time , Then I suddenly realized , There is no first time to write it out , I forgot what doubts I had at that time .
