Musab Gültekin
|
d8252092f7
|
Add duplicate_requests_test.go
|
2021-04-16 14:43:42 +03:00 |
|
Musab Gültekin
|
be4d13c0ef
|
Retry checking refactored using util function.
|
2021-04-14 09:32:42 +03:00 |
|
Musab Gültekin
|
e3d79e2574
|
Added custom logger. Right now, not configurable.
|
2021-04-13 23:36:42 +03:00 |
|
Musab Gültekin
|
cfb16fe1ee
|
Call ErrorFunc on errors. Unexport DoRequestClient and DoRequestChrome
|
2019-12-13 00:03:44 +03:00 |
|
Musab Gültekin
|
7d2fe57bab
|
Added error logging for HTML parser.
|
2019-12-11 13:55:38 +03:00 |
|
Musab Gültekin
|
6645820408
|
Added logging on allowed domains middleware and duplicate requests
|
2019-11-16 20:34:09 +03:00 |
|
Musab Gültekin
|
d19465c44a
|
Robotstxt metrics added.
|
2019-07-08 14:51:54 +03:00 |
|
Musab Gültekin
|
d3c4389c46
|
Retrying support added for chrome. Fixed robots.txt retry issue. Fixed Meta issue
|
2019-07-07 19:50:15 +03:00 |
|
Musab Gültekin
|
42faa92ece
|
Robots.txt support implemented
|
2019-07-06 16:18:03 +03:00 |
|
Musab Gültekin
|
2cab68d2ce
|
Middlewares refactored to multiple files in middleware package.
Extractors removed as they introduce complexity to scraper. Both in learning and developing.
|
2019-07-04 21:04:29 +03:00 |
|