gitea/geziyor
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
71 Commits 1 Branch 0 Tags
Commit Graph

71 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Musab Gültekin
679fd8ab7a Map support added for CSV exporter 2019-06-27 22:39:06 +03:00
Musab Gültekin
8fe194bd10 Added options and tests for exporters. 2019-06-27 16:54:09 +03:00
Musab Gültekin
d20ea47390 Fix Header convertion bug. Map was not canonicalizing keys 2019-06-22 15:04:08 +03:00
Musab Gültekin
02df5aa4e8 Fixed issues on non-trailing URLS on rendered requests 2019-06-22 14:47:12 +03:00
Musab Gültekin
92e7cfefec Fixed README Doc. 2019-06-22 13:13:33 +03:00
Musab Gültekin
a64a262554 HTTP Client can be changed now. Docs updated. 2019-06-22 13:12:05 +03:00
Musab Gültekin
7bc782400c Expvar metrics support added. Metrics refactored to its own package. 2019-06-21 21:37:25 +03:00
Musab Gültekin
88c4b1dd35 Prometheus metrics support added. 2019-06-21 20:05:28 +03:00
Musab Gültekin
141bab0d05 Error handling improved 2019-06-20 10:14:36 +03:00
Musab Gültekin
f88b88986c Delays and logs refactored as middlewares. 2019-06-20 09:54:30 +03:00
Musab Gültekin
514fe2e8d2 Recover system refactored like middleware 2019-06-19 22:45:40 +03:00
Musab Gültekin
c28b228a12 Response header bug fixed for Chrome 2019-06-18 16:37:06 +03:00
Musab Gültekin
ec83a92eb3 Response header support added for Chrome Rendering 2019-06-18 16:26:40 +03:00
Musab Gültekin
217f3c96df Header and native http.Response support added for Chrome rendering 2019-06-18 16:16:29 +03:00
Musab Gültekin
936d157785 Revert "Try parsing HTML even if content-type is empty." 
This reverts commit f384fc2c
 2019-06-18 13:03:00 +03:00
Musab Gültekin
f384fc2c13 Try parsing HTML even if content-type is empty. 2019-06-18 13:00:16 +03:00
Musab Gültekin
4177f10de9 Request creation simplified and basic auth test added. 2019-06-17 13:53:34 +03:00
Musab Gültekin
a5ec28664d Cookies support added. 2019-06-17 13:31:19 +03:00
Musab Gültekin
dd6687f976 Fixed build issue 2019-06-17 12:21:40 +03:00
Musab Gültekin
e50fa3b1dc Response middlewares support implemented. 2019-06-16 18:29:07 +03:00
Musab Gültekin
80383ebd6f Middlewares and some string util functions refactored. Added partial Documentation. 2019-06-16 10:38:03 +03:00
Musab Gültekin
40f673f2e2 Fixed README. More Go versions added for testing 2019-06-15 22:35:51 +03:00
Musab Gültekin
ddff3aee25 Request cancellations support added to Middlewares. 
Some core functions refactored as middlewares.
Fixed race condition in exporting system. Now, only one goroutine will be responsible for exporting. This fixes concurrency issues on writing.
 2019-06-15 22:27:46 +03:00
Musab Gültekin
83a7b9eb87
Merge pull request #4 from NMelis/master 
Create CONTRIBUTING.md
 2019-06-15 18:09:58 +03:00
Musab Gültekin
f65456f18c
Update CONTRIBUTING.md 2019-06-15 18:08:27 +03:00
Musab Gültekin
7b23596a2d Middleware support added. HTML Parsing disable option added. 
Goroutine leaks will be tested using leaktest lib.
 2019-06-15 17:55:40 +03:00
Melis Nurlan
2e29c47acd
Create CONTRIBUTING.md 
could add a description of how to become a contributor?
 2019-06-15 19:43:49 +07:00
Musab Gültekin
4799b0f7b4 Fixed goroutine leaks. Updated travis build 2019-06-14 17:30:49 +03:00
Musab Gültekin
f5b3b0d049 Fixed race conditions on exporters. 
MaxIdleConns limit disabled to support unlimited requests to all hosts.
MaxIdleConnsPerHost limit increased to speed up requests to same host.
 2019-06-14 16:10:36 +03:00
Musab Gültekin
83bfb01856
Merge pull request #3 from isacikgoz/master 
Update README.md
 2019-06-14 15:34:31 +03:00
Musab Gültekin
b2f32b8830
Merge branch 'master' into master 2019-06-14 15:32:36 +03:00
Musab Gültekin
c6ea82447a Travis-CI configuration added. Now, we'll test coverage on every commit. 2019-06-14 15:24:15 +03:00
Musab Gültekin
6caf1effd6 Rendered field exported to support rendered requests on Do function. Data races fixed. 2019-06-14 15:23:56 +03:00
Ibrahim Serdar Acikgoz
7360ffa3c9
Update README.md 2019-06-14 14:57:53 +03:00
Musab Gültekin
1a7d480b36 JS Rendered requests with Chrome support added 2019-06-13 22:08:45 +03:00
Musab Gültekin
76a687e193 Do function refactored 2019-06-13 20:26:07 +03:00
Musab Gültekin
7c43585769
Fixed license copyright 2019-06-13 15:48:02 +03:00
Musab Gültekin
8a6e19a031 New requests on StartRequests func will be made using Geziyor's methods. Not Requests chan 
Options field exported.
 2019-06-13 14:06:37 +03:00
Musab Gültekin
184081d3bf README updated for more advanced usage. Updated tests. 2019-06-12 22:22:01 +03:00
Musab Gültekin
d56ea161a5 Making new requests on StartRequestsFunc is simplified by using channels 2019-06-12 21:54:57 +03:00
Musab Gültekin
f7f4e401e2 Metadata adding on requests support added. StartRequests function implemented. 2019-06-12 21:30:45 +03:00
Musab Gültekin
bd8d58576f Start requests function implemented. 2019-06-12 12:40:38 +03:00
Musab Gültekin
2f6cb06982 Disabling charset detection implemented. 2019-06-12 11:44:31 +03:00
Musab Gültekin
a311a0f998 CSV exporter support added. Not finished for map type. 2019-06-11 20:42:22 +03:00
Musab Gültekin
bbdc3bcacd Exporters made optional, as some scrapers only want to see data in console. 2019-06-11 18:59:37 +03:00
Musab Gültekin
b8305d5e1a Limiting body reading support implemented. 2019-06-11 16:19:30 +03:00
Musab Gültekin
3790295658 Multiple Exporters and custom Exporters support added. 2019-06-11 16:10:49 +03:00
Musab Gültekin
e4e8723426 Callback are now mandatory as almost all the scrapers use it. 2019-06-11 14:24:48 +03:00
Musab Gültekin
ca2414c5c8 Request callbacks added. 
Recover from all panics and continue scraping.
Only parse HTML if response is HTML.
 2019-06-09 21:13:30 +03:00
Musab Gültekin
7abc7a370d Disabling logs support added. 2019-06-09 19:14:46 +03:00