Web Scraping With Scrapy

After reading a reddit thread about interesting scripts people have written to automate things I was inspired to put together a quick web scraping script to check the sales at the LCBO and send me a message (on Telegram) if there is anything interesting.

Back in the day I would have done something like this with BeautifulSoup, but Scrapy is a full-on framework to help build these kinds of tools. It makes it easier to write a web scraping or spider script in a more normalized way and decompose the scraping into a more consise and easier to read functions.

This was also the first time I tried to integrate with Telegram as a bot to send myself messages.  Having built bots for Slack and Facebook Messenger in the past, Telegram’s integration was by far the simplest and quickest to get up and running.  A quick conversation with @BotFather and I had my bot created and a key to connect with.  Using the twx.botapi package I was sending messages in just 3 lines of code.