2 Commits

Author	SHA1	Message	Date
Musab Gültekin	d19465c44a	Robotstxt metrics added.	2019-07-08 14:51:54 +03:00
Musab Gültekin	2cab68d2ce	Middlewares refactored to multiple files in middleware package. ... Extractors removed as they introduce complexity to scraper. Both in learning and developing.	2019-07-04 21:04:29 +03:00