4 Commits

Author	SHA1	Message	Date
Musab Gültekin	d19465c44a	Robotstxt metrics added.	2019-07-08 14:51:54 +03:00
Musab Gültekin	d3c4389c46	Retrying support added for chrome. Fixed robots.txt retry issue. Fixed Meta issue	2019-07-07 19:50:15 +03:00
Musab Gültekin	42faa92ece	Robots.txt support implemented	2019-07-06 16:18:03 +03:00
Musab Gültekin	2cab68d2ce	Middlewares refactored to multiple files in middleware package. ... Extractors removed as they introduce complexity to scraper. Both in learning and developing.	2019-07-04 21:04:29 +03:00