Je veux gratter des titres de film de cette page: https://www.imdb.com/ Liste / ls055386972 / . J'ai écrit le code suivant:
[u'https://www.imdb.com/title/tt0068646/?ref_=ttls_li_tt', u'https://www.imdb.com/title/tt0108052/?ref_=ttls_li_tt', u'https://www.imdb.com/title/tt0050083/?ref_=ttls_li_tt', u'https://www.imdb.com/title/tt0118799/?ref_=ttls_li_tt', u'https://www.imdb.com/title/tt0060196/?ref_=ttls_li_tt',..........]
3 Réponses :
Vous obtenez un 404 car votre URL de départ est incorrecte. Vous devez supprimer la barre oblique suivante dans aussi, votre se référer à la Documentation . start_urls : autorisé_domains est incorrect. Il ne doit contenir que des domaines, pas des urls partiels:
Il ne s'agit toujours pas de ramper
@Debbie Vous devez mettre à jour la question avec la nouvelle sortie ou poser une nouvelle question s'il s'agit d'un problème différent.
même problème. C'est le problème pour lequel j'ai posé la question.
@Debbie alors vous ne l'avez pas réparais. Si vous supprimez le trailing / , vous n'obtiendrez pas l'erreur 404.
Pas sûr de SCHAPY, mais si vous utilisez le code ci-dessous, vous obtenez la sortie souhaitée.
import requests
import lxml
from lxml import html
url = "https://www.imdb.com/list/ls055386972/"
r = requests.get(url)
tree = html.fromstring(r.content)
domain = "https://www.imdb.com/"
movies_list = tree.xpath('//div[@class="lister-item-content"]/h3/a')
for i, movie in enumerate(movies_list, 1):
print(i, movie.text, domain+movie.get('href'))
1 The Godfather https://www.imdb.com//title/tt0068646/?ref_=ttls_li_tt
2 Schindler's List https://www.imdb.com//title/tt0108052/?ref_=ttls_li_tt
3 12 Angry Men https://www.imdb.com//title/tt0050083/?ref_=ttls_li_tt
4 Life Is Beautiful https://www.imdb.com//title/tt0118799/?ref_=ttls_li_tt
5 The Good, the Bad and the Ugly https://www.imdb.com//title/tt0060196/?ref_=ttls_li_tt
6 The Shawshank Redemption https://www.imdb.com//title/tt0111161/?ref_=ttls_li_tt
7 The Pursuit of Happyness https://www.imdb.com//title/tt0454921/?ref_=ttls_li_tt
8 Seven Samurai https://www.imdb.com//title/tt0047478/?ref_=ttls_li_tt
9 The Intouchables https://www.imdb.com//title/tt1675434/?ref_=ttls_li_tt
10 Central Station https://www.imdb.com//title/tt0140888/?ref_=ttls_li_tt
11 Requiem for a Dream https://www.imdb.com//title/tt0180093/?ref_=ttls_li_tt
12 A Beautiful Mind https://www.imdb.com//title/tt0268978/?ref_=ttls_li_tt
13 Hachi: A Dog's Tale https://www.imdb.com//title/tt1028532/?ref_=ttls_li_tt
14 Taken https://www.imdb.com//title/tt0936501/?ref_=ttls_li_tt
15 My Sassy Girl https://www.imdb.com//title/tt0293715/?ref_=ttls_li_tt
16 Amores Perros https://www.imdb.com//title/tt0245712/?ref_=ttls_li_tt
17 The Shining https://www.imdb.com//title/tt0081505/?ref_=ttls_li_tt
18 Apocalypto https://www.imdb.com//title/tt0472043/?ref_=ttls_li_tt
19 Gladiator https://www.imdb.com//title/tt0172495/?ref_=ttls_li_tt
20 Cast Away https://www.imdb.com//title/tt0162222/?ref_=ttls_li_tt
21 The Dark Knight https://www.imdb.com//title/tt0468569/?ref_=ttls_li_tt
22 The Pianist https://www.imdb.com//title/tt0253474/?ref_=ttls_li_tt
23 Titanic https://www.imdb.com//title/tt0120338/?ref_=ttls_li_tt
24 3-Iron https://www.imdb.com//title/tt0423866/?ref_=ttls_li_tt
25 Braveheart https://www.imdb.com//title/tt0112573/?ref_=ttls_li_tt
26 It's a Wonderful Life https://www.imdb.com//title/tt0038650/?ref_=ttls_li_tt
27 Spring, Summer, Fall, Winter... and Spring https://www.imdb.com//title/tt0374546/?ref_=ttls_li_tt
28 Alien https://www.imdb.com//title/tt0078748/?ref_=ttls_li_tt
29 Memories of Murder https://www.imdb.com//title/tt0353969/?ref_=ttls_li_tt
30 The Return https://www.imdb.com//title/tt0376968/?ref_=ttls_li_tt
31 I Saw the Devil https://www.imdb.com//title/tt1588170/?ref_=ttls_li_tt
32 Children of Heaven https://www.imdb.com//title/tt0118849/?ref_=ttls_li_tt
33 A Separation https://www.imdb.com//title/tt1832382/?ref_=ttls_li_tt
34 The Sixth Sense https://www.imdb.com//title/tt0167404/?ref_=ttls_li_tt
35 A Moment to Remember https://www.imdb.com//title/tt0428870/?ref_=ttls_li_tt
36 Departures https://www.imdb.com//title/tt1069238/?ref_=ttls_li_tt
37 Wo de fu qin mu qin https://www.imdb.com//title/tt0235060/?ref_=ttls_li_tt
38 Saving Private Ryan https://www.imdb.com//title/tt0120815/?ref_=ttls_li_tt
39 The Bridge on the River Kwai https://www.imdb.com//title/tt0050212/?ref_=ttls_li_tt
40 Ben-Hur https://www.imdb.com//title/tt0052618/?ref_=ttls_li_tt
41 The Exorcist https://www.imdb.com//title/tt0070047/?ref_=ttls_li_tt
42 The Secret in Their Eyes https://www.imdb.com//title/tt1305806/?ref_=ttls_li_tt
43 Léon: The Professional https://www.imdb.com//title/tt0110413/?ref_=ttls_li_tt
44 The Green Mile https://www.imdb.com//title/tt0120689/?ref_=ttls_li_tt
45 Gran Torino https://www.imdb.com//title/tt1205489/?ref_=ttls_li_tt
46 Kill Bill: Vol. 1 https://www.imdb.com//title/tt0266697/?ref_=ttls_li_tt
47 Jurassic Park https://www.imdb.com//title/tt0107290/?ref_=ttls_li_tt
48 Terminator 2: Judgment Day https://www.imdb.com//title/tt0103064/?ref_=ttls_li_tt
49 Back to the Future https://www.imdb.com//title/tt0088763/?ref_=ttls_li_tt
50 Finding Nemo https://www.imdb.com//title/tt0266543/?ref_=ttls_li_tt
Vous n'avez pas analysé l'URL de démarrage à la fonction Payse, ci-dessous est du code de travail.
class MoviesSpider(BaseSpider):
name = 'movies' #name of the spider
allowed_domains = ['imdb.com']
start_url = 'http://imdb.com/list/ls055386972/'
def __init__(self):
super(MoviesSpider, self).__init__()
def start_requests(self):
yield Request(self.start_url, callback=self.parse, headers=self.headers)
def parse(self, response):
#events = response.xpath('//*[@property="url"]/@href').extract()
links = response.xpath('//h3[@class]/a/@href').extract()
final_links = []
for link in links:
final_link = 'http://www.imdb.com' + link
final_links.append(final_link)
for final_link in final_links:
absolute_url = response.urljoin(final_link)
yield Request(absolute_url, callback = self.parse_movies)
#process next page url
#next_page_url = response.xpath('//a[text() = "Next"]/@href').extract_first()
#absolute_next_page_url = response.urljoin(next_page_url)
#yield Request(absolute_next_page_url)
def parse_movies(self, response):
title = response.xpath('//div[@class = "title_wrapper"]/h1[@class]/text()').extract_first()
yield{
'title': title,
}