3 Commits

Author	SHA1	Message	Date
Musab Gültekin	d19465c44a	Robotstxt metrics added.	2019-07-08 14:51:54 +03:00
Musab Gültekin	d3c4389c46	Retrying support added for chrome. Fixed robots.txt retry issue. Fixed Meta issue	2019-07-07 19:50:15 +03:00
Musab Gültekin	42faa92ece	Robots.txt support implemented	2019-07-06 16:18:03 +03:00