Musab Gültekin
|
6645820408
|
Added logging on allowed domains middleware and duplicate requests
|
2019-11-16 20:34:09 +03:00 |
|
Musab Gültekin
|
d19465c44a
|
Robotstxt metrics added.
|
2019-07-08 14:51:54 +03:00 |
|
Musab Gültekin
|
d3c4389c46
|
Retrying support added for chrome. Fixed robots.txt retry issue. Fixed Meta issue
|
2019-07-07 19:50:15 +03:00 |
|
Musab Gültekin
|
42faa92ece
|
Robots.txt support implemented
|
2019-07-06 16:18:03 +03:00 |
|
Musab Gültekin
|
2cab68d2ce
|
Middlewares refactored to multiple files in middleware package.
Extractors removed as they introduce complexity to scraper. Both in learning and developing.
|
2019-07-04 21:04:29 +03:00 |
|