Gfrktyl

I've created a script using Python in association with Scrapy to parse the movie names and its years spread across multiple pages from a torrent site. My goal here is to write the parsed data in a CSV file other than using the built-in command provided by Scrapy, because when I do this:

scrapy crawl  torrentdata -o outputfile.csv -t csv

I get a blank line in every alternate row in the CSV file.

However, I thought to go in a slightly different way to achieve the same thing. Now, I get a data-laden CSV file in the right format when I run the following script. Most importantly I made use of a with statement while creating a CSV file so that when the writing is done the file gets automatically closed. I used crawlerprocess to execute the script from within an IDE.

My question: Isn't it a better idea for me to follow the way I tried below?

This is the working script:

import scrapy

from scrapy.crawler import CrawlerProcess

import csv



class TorrentSpider(scrapy.Spider):

    name = "torrentdata"

    start_urls = ["https://yts.am/browse-movies?page={}".format(page) for page in range(2,20)] #get something within list

    itemlist = 



    def parse(self, response):

        for record in response.css('.browse-movie-bottom'):

            items = {}

            items["Name"] = record.css('.browse-movie-title::text').extract_first(default='')

            items["Year"] = record.css('.browse-movie-year::text').extract_first(default='')

            self.itemlist.append(items)



        with open("outputfile.csv","w", newline="") as f:

            writer = csv.DictWriter(f,['Name','Year'])

            writer.writeheader()

            for data in self.itemlist:

                writer.writerow(data)



c = CrawlerProcess({

    'USER_AGENT': 'Mozilla/5.0',   

})

c.crawl(TorrentSpider)

c.start() 

By putting the CSV exporting logic into the spider itself, you are re-inventing the wheel and not using all the advantages of Scrapy and its components and, also, making the crawling slower as you are writing to disk in the crawling stage every time the callback is triggered.

As you mentioned, the CSV exporter is built-in, you just need to yield/return items from the parse() callback:

import scrapy





class TorrentSpider(scrapy.Spider):

    name = "torrentdata"

    start_urls = ["https://yts.am/browse-movies?page={}".format(page) for page in range(2,20)] #get something within list



    def parse(self, response):

        for record in response.css('.browse-movie-bottom'):

            yield {

                "Name": record.css('.browse-movie-title::text').extract_first(default=''),

                "Year": record.css('.browse-movie-year::text').extract_first(default='')

            }

Then, by running:

scrapy runspider spider.py -o outputfile.csv -t csv

(or the crawl command)

you would have the following in the outputfile.csv:

Name,Year

"Faith, Love & Chocolate",2018

Bennett's Song,2018

...

Tender Mercies,1983

You Might Be the Killer,2018

Although I'm not an expert on this, I thought to come up with a solution which I've been following quite some time.
Making use of signals might be a wise attempt here. When the scraping process is done, the spider_closed() method is invoked and thus the DictWriter() will be open once and when the writing is finished, it will be closed automatically because of the with statement. That said there is hardly any chance for your script to be slower, if you can get rid of Disk I/O issues.

The following script represents what I told you so far:

import scrapy

from scrapy.crawler import CrawlerProcess

from scrapy import signals

import csv



class TorrentSpider(scrapy.Spider):

    name = "torrentdata"

    start_urls = ["https://yts.am/browse-movies?page={}".format(page) for page in range(2,10)] #get something within list

    itemlist = 



    @classmethod

    def from_crawler(cls, crawler):

        spider = super().from_crawler(crawler)

        crawler.signals.connect(spider.spider_closed, signals.spider_closed)

        return spider



    def spider_closed(self):

        with open("outputfile.csv","w", newline="") as f:

            writer = csv.DictWriter(f,['Name','Year'])

            writer.writeheader()

            for data in self.itemlist:

                writer.writerow(data)



    def parse(self, response):

        for record in response.css('.browse-movie-bottom'):

            items = {}

            items["Name"] = record.css('.browse-movie-title::text').extract_first(default='')

            items["Year"] = record.css('.browse-movie-year::text').extract_first(default='')

            self.itemlist.append(items)



c = CrawlerProcess({

    'USER_AGENT': 'Mozilla/5.0',   

})

c.crawl(TorrentSpider)

c.start()