2 Commits

Author	SHA1	Message	Date
Musab Gültekin	42faa92ece	Robots.txt support implemented	2019-07-06 16:18:03 +03:00
Musab Gültekin	2cab68d2ce	Middlewares refactored to multiple files in middleware package. ... Extractors removed as they introduce complexity to scraper. Both in learning and developing.	2019-07-04 21:04:29 +03:00