gitea/geziyor
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
66 Commits 1 Branch 0 Tags
a64a262554bbb1fc76bbfeba758d1ffba882dcf4
Commit Graph
58 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Musab Gültekin a64a262554 HTTP Client can be changed now. Docs updated. 2019-06-22 13:12:05 +03:00
Musab Gültekin 7bc782400c Expvar metrics support added. Metrics refactored to its own package. 2019-06-21 21:37:25 +03:00
Musab Gültekin 88c4b1dd35 Prometheus metrics support added. 2019-06-21 20:05:28 +03:00
Musab Gültekin 141bab0d05 Error handling improved 2019-06-20 10:14:36 +03:00
Musab Gültekin f88b88986c Delays and logs refactored as middlewares. 2019-06-20 09:54:30 +03:00
Musab Gültekin 514fe2e8d2 Recover system refactored like middleware 2019-06-19 22:45:40 +03:00
Musab Gültekin c28b228a12 Response header bug fixed for Chrome 2019-06-18 16:37:06 +03:00
Musab Gültekin ec83a92eb3 Response header support added for Chrome Rendering 2019-06-18 16:26:40 +03:00
Musab Gültekin 217f3c96df Header and native http.Response support added for Chrome rendering 2019-06-18 16:16:29 +03:00
Musab Gültekin 936d157785 Revert "Try parsing HTML even if content-type is empty." 
This reverts commit f384fc2c
 2019-06-18 13:03:00 +03:00
Musab Gültekin f384fc2c13 Try parsing HTML even if content-type is empty. 2019-06-18 13:00:16 +03:00
Musab Gültekin 4177f10de9 Request creation simplified and basic auth test added. 2019-06-17 13:53:34 +03:00
Musab Gültekin a5ec28664d Cookies support added. 2019-06-17 13:31:19 +03:00
Musab Gültekin dd6687f976 Fixed build issue 2019-06-17 12:21:40 +03:00
Musab Gültekin e50fa3b1dc Response middlewares support implemented. 2019-06-16 18:29:07 +03:00
Musab Gültekin 80383ebd6f Middlewares and some string util functions refactored. Added partial Documentation. 2019-06-16 10:38:03 +03:00
Musab Gültekin 40f673f2e2 Fixed README. More Go versions added for testing 2019-06-15 22:35:51 +03:00
Musab Gültekin ddff3aee25 Request cancellations support added to Middlewares. 
Some core functions refactored as middlewares.
Fixed race condition in exporting system. Now, only one goroutine will be responsible for exporting. This fixes concurrency issues on writing.
 2019-06-15 22:27:46 +03:00
Musab Gültekin 7b23596a2d Middleware support added. HTML Parsing disable option added. 
Goroutine leaks will be tested using leaktest lib.
 2019-06-15 17:55:40 +03:00
Musab Gültekin 4799b0f7b4 Fixed goroutine leaks. Updated travis build 2019-06-14 17:30:49 +03:00
Musab Gültekin f5b3b0d049 Fixed race conditions on exporters. 
MaxIdleConns limit disabled to support unlimited requests to all hosts.
MaxIdleConnsPerHost limit increased to speed up requests to same host.
 2019-06-14 16:10:36 +03:00
Musab Gültekin c6ea82447a Travis-CI configuration added. Now, we'll test coverage on every commit. 2019-06-14 15:24:15 +03:00
Musab Gültekin 6caf1effd6 Rendered field exported to support rendered requests on Do function. Data races fixed. 2019-06-14 15:23:56 +03:00
Musab Gültekin 1a7d480b36 JS Rendered requests with Chrome support added 2019-06-13 22:08:45 +03:00
Musab Gültekin 76a687e193 Do function refactored 2019-06-13 20:26:07 +03:00
Musab Gültekin 8a6e19a031 New requests on StartRequests func will be made using Geziyor's methods. Not Requests chan 
Options field exported.
 2019-06-13 14:06:37 +03:00
Musab Gültekin 184081d3bf README updated for more advanced usage. Updated tests. 2019-06-12 22:22:01 +03:00
Musab Gültekin d56ea161a5 Making new requests on StartRequestsFunc is simplified by using channels 2019-06-12 21:54:57 +03:00
Musab Gültekin f7f4e401e2 Metadata adding on requests support added. StartRequests function implemented. 2019-06-12 21:30:45 +03:00
Musab Gültekin bd8d58576f Start requests function implemented. 2019-06-12 12:40:38 +03:00
Musab Gültekin 2f6cb06982 Disabling charset detection implemented. 2019-06-12 11:44:31 +03:00
Musab Gültekin a311a0f998 CSV exporter support added. Not finished for map type. 2019-06-11 20:42:22 +03:00
Musab Gültekin bbdc3bcacd Exporters made optional, as some scrapers only want to see data in console. 2019-06-11 18:59:37 +03:00
Musab Gültekin b8305d5e1a Limiting body reading support implemented. 2019-06-11 16:19:30 +03:00
Musab Gültekin 3790295658 Multiple Exporters and custom Exporters support added. 2019-06-11 16:10:49 +03:00
Musab Gültekin e4e8723426 Callback are now mandatory as almost all the scrapers use it. 2019-06-11 14:24:48 +03:00
Musab Gültekin ca2414c5c8 Request callbacks added. 
Recover from all panics and continue scraping.
Only parse HTML if response is HTML.
 2019-06-09 21:13:30 +03:00
Musab Gültekin 7abc7a370d Disabling logs support added. 2019-06-09 19:14:46 +03:00
Musab Gültekin b973c1c064 Request delays support added 2019-06-09 14:24:53 +03:00
Musab Gültekin 2263108838 User-Agent change support added. 2019-06-09 13:43:17 +03:00
Musab Gültekin 9263877339 Exporting data all types support added. 2019-06-09 13:22:20 +03:00
Musab Gültekin d967555b62 Global and Domain Concurrency limit implemented. Updated README 2019-06-09 11:53:40 +03:00
Musab Gültekin a9aaf86df3 Automatic determining response and decoding it. 2019-06-09 10:46:32 +03:00
Musab Gültekin f2fba85266 Timeout support added 2019-06-08 20:36:43 +03:00
Musab Gültekin 2e3bd18430 Options refactored to its own file. Timeout increased to 60 sec 2019-06-08 20:36:43 +03:00
Musab Gültekin b90908066b Head API added. Opt renamed to Options. Tests updated. More documentation added. 2019-06-08 20:36:43 +03:00
Musab Gültekin 815ae7eec5 Do request support added. Updated docs. 2019-06-08 19:45:48 +03:00
Musab Gültekin 54c7d3550f Gezer renamed to Geziyor 2019-06-08 17:14:10 +03:00
Musab Gültekin c525e0d7d0 Don't visit already visited URLs. Update README 2019-06-08 17:04:00 +03:00
Musab Gültekin edbddf74d8 Cache added to options 2019-06-08 16:01:31 +03:00
Musab Gültekin ca197ff06a Caching added. 
JSON File export will append, not truncate.
 2019-06-08 15:29:09 +03:00
Musab Gültekin 9e61a96412 Sync and Async requests support added by using go keyword. 2019-06-07 21:31:18 +03:00
Musab Gültekin ee7d498f22 Allowed domains support added 2019-06-07 21:02:31 +03:00
Musab Gültekin 5271095c8e Export chan added. We can now export data by sending to this chan. 2019-06-07 20:52:31 +03:00
Musab Gültekin e58b08cbd6 Relative URL handling added to Response type. 2019-06-07 15:44:14 +03:00
Musab Gültekin 944bd3bada Options used instead of direct parameter passing. 
Tests updated.
 2019-06-06 23:49:42 +03:00
Musab Gültekin 6358b87472 Use parse function to parse responses, instead of channels. 
Parse response as HTML Document using goquery.
Added simple README.
 2019-06-06 22:48:57 +03:00
Musab Gültekin 1c96048082 Initial commit 2019-06-06 17:11:19 +03:00