gitea/geziyor
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
51 Commits 1 Branch 0 Tags
Commit Graph

30 Commits

Author SHA1 Message Date
Musab Gültekin
80383ebd6f Middlewares and some string util functions refactored. Added partial Documentation. 2019-06-16 10:38:03 +03:00
Musab Gültekin
ddff3aee25 Request cancellations support added to Middlewares. 
Some core functions refactored as middlewares.
Fixed race condition in exporting system. Now, only one goroutine will be responsible for exporting. This fixes concurrency issues on writing.
 2019-06-15 22:27:46 +03:00
Musab Gültekin
7b23596a2d Middleware support added. HTML Parsing disable option added. 
Goroutine leaks will be tested using leaktest lib.
 2019-06-15 17:55:40 +03:00
Musab Gültekin
4799b0f7b4 Fixed goroutine leaks. Updated travis build 2019-06-14 17:30:49 +03:00
Musab Gültekin
f5b3b0d049 Fixed race conditions on exporters. 
MaxIdleConns limit disabled to support unlimited requests to all hosts.
MaxIdleConnsPerHost limit increased to speed up requests to same host.
 2019-06-14 16:10:36 +03:00
Musab Gültekin
6caf1effd6 Rendered field exported to support rendered requests on Do function. Data races fixed. 2019-06-14 15:23:56 +03:00
Musab Gültekin
1a7d480b36 JS Rendered requests with Chrome support added 2019-06-13 22:08:45 +03:00
Musab Gültekin
76a687e193 Do function refactored 2019-06-13 20:26:07 +03:00
Musab Gültekin
8a6e19a031 New requests on StartRequests func will be made using Geziyor's methods. Not Requests chan 
Options field exported.
 2019-06-13 14:06:37 +03:00
Musab Gültekin
d56ea161a5 Making new requests on StartRequestsFunc is simplified by using channels 2019-06-12 21:54:57 +03:00
Musab Gültekin
f7f4e401e2 Metadata adding on requests support added. StartRequests function implemented. 2019-06-12 21:30:45 +03:00
Musab Gültekin
bd8d58576f Start requests function implemented. 2019-06-12 12:40:38 +03:00
Musab Gültekin
2f6cb06982 Disabling charset detection implemented. 2019-06-12 11:44:31 +03:00
Musab Gültekin
a311a0f998 CSV exporter support added. Not finished for map type. 2019-06-11 20:42:22 +03:00
Musab Gültekin
bbdc3bcacd Exporters made optional, as some scrapers only want to see data in console. 2019-06-11 18:59:37 +03:00
Musab Gültekin
b8305d5e1a Limiting body reading support implemented. 2019-06-11 16:19:30 +03:00
Musab Gültekin
3790295658 Multiple Exporters and custom Exporters support added. 2019-06-11 16:10:49 +03:00
Musab Gültekin
e4e8723426 Callback are now mandatory as almost all the scrapers use it. 2019-06-11 14:24:48 +03:00
Musab Gültekin
ca2414c5c8 Request callbacks added. 
Recover from all panics and continue scraping.
Only parse HTML if response is HTML.
 2019-06-09 21:13:30 +03:00
Musab Gültekin
7abc7a370d Disabling logs support added. 2019-06-09 19:14:46 +03:00
Musab Gültekin
b973c1c064 Request delays support added 2019-06-09 14:24:53 +03:00
Musab Gültekin
2263108838 User-Agent change support added. 2019-06-09 13:43:17 +03:00
Musab Gültekin
9263877339 Exporting data all types support added. 2019-06-09 13:22:20 +03:00
Musab Gültekin
d967555b62 Global and Domain Concurrency limit implemented. Updated README 2019-06-09 11:53:40 +03:00
Musab Gültekin
a9aaf86df3 Automatic determining response and decoding it. 2019-06-09 10:46:32 +03:00
Musab Gültekin
f2fba85266 Timeout support added 2019-06-08 20:36:43 +03:00
Musab Gültekin
2e3bd18430 Options refactored to its own file. Timeout increased to 60 sec 2019-06-08 20:36:43 +03:00
Musab Gültekin
b90908066b Head API added. Opt renamed to Options. Tests updated. More documentation added. 2019-06-08 20:36:43 +03:00
Musab Gültekin
815ae7eec5 Do request support added. Updated docs. 2019-06-08 19:45:48 +03:00
Musab Gültekin
54c7d3550f Gezer renamed to Geziyor 2019-06-08 17:14:10 +03:00