Musab Gültekin
80383ebd6f
Middlewares and some string util functions refactored. Added partial Documentation.
2019-06-16 10:38:03 +03:00
Musab Gültekin
ddff3aee25
Request cancellations support added to Middlewares.
...
Some core functions refactored as middlewares.
Fixed race condition in exporting system. Now, only one goroutine will be responsible for exporting. This fixes concurrency issues on writing.
2019-06-15 22:27:46 +03:00
Musab Gültekin
7b23596a2d
Middleware support added. HTML Parsing disable option added.
...
Goroutine leaks will be tested using leaktest lib.
2019-06-15 17:55:40 +03:00
Musab Gültekin
6caf1effd6
Rendered field exported to support rendered requests on Do function. Data races fixed.
2019-06-14 15:23:56 +03:00
Musab Gültekin
1a7d480b36
JS Rendered requests with Chrome support added
2019-06-13 22:08:45 +03:00
Musab Gültekin
8a6e19a031
New requests on StartRequests func will be made using Geziyor's methods. Not Requests chan
...
Options field exported.
2019-06-13 14:06:37 +03:00
Musab Gültekin
184081d3bf
README updated for more advanced usage. Updated tests.
2019-06-12 22:22:01 +03:00
Musab Gültekin
d56ea161a5
Making new requests on StartRequestsFunc is simplified by using channels
2019-06-12 21:54:57 +03:00
Musab Gültekin
f7f4e401e2
Metadata adding on requests support added. StartRequests function implemented.
2019-06-12 21:30:45 +03:00
Musab Gültekin
bd8d58576f
Start requests function implemented.
2019-06-12 12:40:38 +03:00
Musab Gültekin
a311a0f998
CSV exporter support added. Not finished for map type.
2019-06-11 20:42:22 +03:00
Musab Gültekin
bbdc3bcacd
Exporters made optional, as some scrapers only want to see data in console.
2019-06-11 18:59:37 +03:00
Musab Gültekin
e4e8723426
Callback are now mandatory as almost all the scrapers use it.
2019-06-11 14:24:48 +03:00
Musab Gültekin
ca2414c5c8
Request callbacks added.
...
Recover from all panics and continue scraping.
Only parse HTML if response is HTML.
2019-06-09 21:13:30 +03:00
Musab Gültekin
b973c1c064
Request delays support added
2019-06-09 14:24:53 +03:00
Musab Gültekin
9263877339
Exporting data all types support added.
2019-06-09 13:22:20 +03:00
Musab Gültekin
d967555b62
Global and Domain Concurrency limit implemented. Updated README
2019-06-09 11:53:40 +03:00
Musab Gültekin
b90908066b
Head API added. Opt renamed to Options. Tests updated. More documentation added.
2019-06-08 20:36:43 +03:00
Musab Gültekin
815ae7eec5
Do request support added. Updated docs.
2019-06-08 19:45:48 +03:00
Musab Gültekin
54c7d3550f
Gezer renamed to Geziyor
2019-06-08 17:14:10 +03:00
