Musab Gültekin
|
738852f932
|
Add custom actions for rendered requests & Fix not closing bug
|
2022-04-29 03:05:31 +03:00 |
|
Musab Gültekin
|
97ecb7f118
|
Proxy support
|
2021-09-24 16:15:20 +03:00 |
|
Musab Gültekin
|
fbee722a38
|
Rate limiting per second implemented
|
2021-04-16 15:31:31 +03:00 |
|
Musab Gültekin
|
cfb16fe1ee
|
Call ErrorFunc on errors. Unexport DoRequestClient and DoRequestChrome
|
2019-12-13 00:03:44 +03:00 |
|
Musab Gültekin
|
0e5230eac8
|
Remote endpoint support added for js rendered requests. Geziyor is beta now.
|
2019-08-05 15:14:47 +03:00 |
|
Musab Gültekin
|
90d2be2210
|
Caching policies added.
We used httpcache library to implement this. As it was not possible to support different policies, I mostly copied and modified it.
|
2019-07-07 12:18:40 +03:00 |
|
Musab Gültekin
|
42faa92ece
|
Robots.txt support implemented
|
2019-07-06 16:18:03 +03:00 |
|
Musab Gültekin
|
2cab68d2ce
|
Middlewares refactored to multiple files in middleware package.
Extractors removed as they introduce complexity to scraper. Both in learning and developing.
|
2019-07-04 21:04:29 +03:00 |
|
Musab Gültekin
|
9adff75509
|
Retry requests support implemented for client.
|
2019-07-04 13:36:10 +03:00 |
|
Musab Gültekin
|
4ab7cfd904
|
Exporter and Extractor interfaces moved to its own package for simplicity of main Geziyor package
|
2019-07-02 13:22:23 +03:00 |
|
Musab Gültekin
|
c0dd0393e6
|
Maximum redirection option added. Performance improvement on exports. Duplicate requests only checked on GET requests.
|
2019-07-01 15:44:28 +03:00 |
|
Musab Gültekin
|
bd6466a5f2
|
http package renamed to client to reduce cunfusion
|
2019-06-29 14:18:31 +03:00 |
|
Musab Gültekin
|
1e109c555d
|
Request and response moved to http package
|
2019-06-29 13:36:39 +03:00 |
|
Musab Gültekin
|
b000581c3d
|
Extractors implemented. Exporters name simplified. README Updated for extracting data. Removed go 1.11 support
|
2019-06-28 13:00:30 +03:00 |
|
Musab Gültekin
|
7bc782400c
|
Expvar metrics support added. Metrics refactored to its own package.
|
2019-06-21 21:37:25 +03:00 |
|
Musab Gültekin
|
a5ec28664d
|
Cookies support added.
|
2019-06-17 13:31:19 +03:00 |
|
Musab Gültekin
|
e50fa3b1dc
|
Response middlewares support implemented.
|
2019-06-16 18:29:07 +03:00 |
|
Musab Gültekin
|
80383ebd6f
|
Middlewares and some string util functions refactored. Added partial Documentation.
|
2019-06-16 10:38:03 +03:00 |
|
Musab Gültekin
|
7b23596a2d
|
Middleware support added. HTML Parsing disable option added.
Goroutine leaks will be tested using leaktest lib.
|
2019-06-15 17:55:40 +03:00 |
|
Musab Gültekin
|
1a7d480b36
|
JS Rendered requests with Chrome support added
|
2019-06-13 22:08:45 +03:00 |
|
Musab Gültekin
|
d56ea161a5
|
Making new requests on StartRequestsFunc is simplified by using channels
|
2019-06-12 21:54:57 +03:00 |
|
Musab Gültekin
|
f7f4e401e2
|
Metadata adding on requests support added. StartRequests function implemented.
|
2019-06-12 21:30:45 +03:00 |
|
Musab Gültekin
|
bd8d58576f
|
Start requests function implemented.
|
2019-06-12 12:40:38 +03:00 |
|
Musab Gültekin
|
2f6cb06982
|
Disabling charset detection implemented.
|
2019-06-12 11:44:31 +03:00 |
|
Musab Gültekin
|
b8305d5e1a
|
Limiting body reading support implemented.
|
2019-06-11 16:19:30 +03:00 |
|
Musab Gültekin
|
3790295658
|
Multiple Exporters and custom Exporters support added.
|
2019-06-11 16:10:49 +03:00 |
|
Musab Gültekin
|
7abc7a370d
|
Disabling logs support added.
|
2019-06-09 19:14:46 +03:00 |
|
Musab Gültekin
|
b973c1c064
|
Request delays support added
|
2019-06-09 14:24:53 +03:00 |
|
Musab Gültekin
|
2263108838
|
User-Agent change support added.
|
2019-06-09 13:43:17 +03:00 |
|
Musab Gültekin
|
d967555b62
|
Global and Domain Concurrency limit implemented. Updated README
|
2019-06-09 11:53:40 +03:00 |
|
Musab Gültekin
|
f2fba85266
|
Timeout support added
|
2019-06-08 20:36:43 +03:00 |
|
Musab Gültekin
|
2e3bd18430
|
Options refactored to its own file. Timeout increased to 60 sec
|
2019-06-08 20:36:43 +03:00 |
|