gitea/geziyor
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Commit Graph

  • 217f3c96df Header and native http.Response support added for Chrome rendering Musab Gültekin 2019-06-18 16:16:29 +03:00
  • 936d157785 Revert "Try parsing HTML even if content-type is empty." Musab Gültekin 2019-06-18 13:03:00 +03:00
  • f384fc2c13 Try parsing HTML even if content-type is empty. Musab Gültekin 2019-06-18 13:00:16 +03:00
  • 4177f10de9 Request creation simplified and basic auth test added. Musab Gültekin 2019-06-17 13:53:34 +03:00
  • a5ec28664d Cookies support added. Musab Gültekin 2019-06-17 13:31:19 +03:00
  • dd6687f976 Fixed build issue Musab Gültekin 2019-06-17 12:21:40 +03:00
  • e50fa3b1dc Response middlewares support implemented. Musab Gültekin 2019-06-16 18:29:07 +03:00
  • 80383ebd6f Middlewares and some string util functions refactored. Added partial Documentation. Musab Gültekin 2019-06-16 10:38:03 +03:00
  • 40f673f2e2 Fixed README. More Go versions added for testing Musab Gültekin 2019-06-15 22:35:51 +03:00
  • ddff3aee25 Request cancellations support added to Middlewares. Some core functions refactored as middlewares. Fixed race condition in exporting system. Now, only one goroutine will be responsible for exporting. This fixes concurrency issues on writing. Musab Gültekin 2019-06-15 22:27:46 +03:00
  • 83a7b9eb87 Merge pull request #4 from NMelis/master Musab Gültekin 2019-06-15 18:09:58 +03:00
  • f65456f18c Update CONTRIBUTING.md Musab Gültekin 2019-06-15 18:08:27 +03:00
  • 7b23596a2d Middleware support added. HTML Parsing disable option added. Goroutine leaks will be tested using leaktest lib. Musab Gültekin 2019-06-15 17:55:40 +03:00
  • 2e29c47acd Create CONTRIBUTING.md Melis Nurlan 2019-06-15 19:43:49 +07:00
  • 4799b0f7b4 Fixed goroutine leaks. Updated travis build Musab Gültekin 2019-06-14 17:30:49 +03:00
  • f5b3b0d049 Fixed race conditions on exporters. MaxIdleConns limit disabled to support unlimited requests to all hosts. MaxIdleConnsPerHost limit increased to speed up requests to same host. Musab Gültekin 2019-06-14 16:10:24 +03:00
  • 83bfb01856 Merge pull request #3 from isacikgoz/master Musab Gültekin 2019-06-14 15:34:31 +03:00
  • b2f32b8830 Merge branch 'master' into master Musab Gültekin 2019-06-14 15:32:36 +03:00
  • c6ea82447a Travis-CI configuration added. Now, we'll test coverage on every commit. Musab Gültekin 2019-06-14 15:24:15 +03:00
  • 6caf1effd6 Rendered field exported to support rendered requests on Do function. Data races fixed. Musab Gültekin 2019-06-14 15:23:56 +03:00
  • 7360ffa3c9 Update README.md Ibrahim Serdar Acikgoz 2019-06-14 14:57:53 +03:00
  • 1a7d480b36 JS Rendered requests with Chrome support added Musab Gültekin 2019-06-13 22:08:45 +03:00
  • 76a687e193 Do function refactored Musab Gültekin 2019-06-13 20:12:59 +03:00
  • 7c43585769 Fixed license copyright Musab Gültekin 2019-06-13 15:48:02 +03:00
  • 8a6e19a031 New requests on StartRequests func will be made using Geziyor's methods. Not Requests chan Options field exported. Musab Gültekin 2019-06-13 14:06:37 +03:00
  • 184081d3bf README updated for more advanced usage. Updated tests. Musab Gültekin 2019-06-12 22:22:01 +03:00
  • d56ea161a5 Making new requests on StartRequestsFunc is simplified by using channels Musab Gültekin 2019-06-12 21:54:57 +03:00
  • f7f4e401e2 Metadata adding on requests support added. StartRequests function implemented. Musab Gültekin 2019-06-12 21:30:45 +03:00
  • bd8d58576f Start requests function implemented. Musab Gültekin 2019-06-12 12:40:38 +03:00
  • 2f6cb06982 Disabling charset detection implemented. Musab Gültekin 2019-06-12 11:44:31 +03:00
  • a311a0f998 CSV exporter support added. Not finished for map type. Musab Gültekin 2019-06-11 20:42:22 +03:00
  • bbdc3bcacd Exporters made optional, as some scrapers only want to see data in console. Musab Gültekin 2019-06-11 18:59:37 +03:00
  • b8305d5e1a Limiting body reading support implemented. Musab Gültekin 2019-06-11 16:19:30 +03:00
  • 3790295658 Multiple Exporters and custom Exporters support added. Musab Gültekin 2019-06-11 16:10:49 +03:00
  • e4e8723426 Callback are now mandatory as almost all the scrapers use it. Musab Gültekin 2019-06-11 14:24:48 +03:00
  • ca2414c5c8 Request callbacks added. Recover from all panics and continue scraping. Only parse HTML if response is HTML. Musab Gültekin 2019-06-09 21:13:30 +03:00
  • 7abc7a370d Disabling logs support added. Musab Gültekin 2019-06-09 19:14:46 +03:00
  • b973c1c064 Request delays support added Musab Gültekin 2019-06-09 14:24:53 +03:00
  • 2263108838 User-Agent change support added. Musab Gültekin 2019-06-09 13:43:17 +03:00
  • 9263877339 Exporting data all types support added. Musab Gültekin 2019-06-09 13:22:20 +03:00
  • d967555b62 Global and Domain Concurrency limit implemented. Updated README Musab Gültekin 2019-06-09 11:53:40 +03:00
  • a9aaf86df3 Automatic determining response and decoding it. Musab Gültekin 2019-06-09 10:46:32 +03:00
  • f2fba85266 Timeout support added Musab Gültekin 2019-06-08 20:36:11 +03:00
  • 2e3bd18430 Options refactored to its own file. Timeout increased to 60 sec Musab Gültekin 2019-06-08 20:27:45 +03:00
  • b90908066b Head API added. Opt renamed to Options. Tests updated. More documentation added. Musab Gültekin 2019-06-08 20:13:16 +03:00
  • 95d97436bf Create LICENSE.txt Musab Gültekin 2019-06-08 19:59:49 +03:00
  • 815ae7eec5 Do request support added. Updated docs. Musab Gültekin 2019-06-08 19:45:48 +03:00
  • 54c7d3550f Gezer renamed to Geziyor Musab Gültekin 2019-06-08 17:14:10 +03:00
  • c525e0d7d0 Don't visit already visited URLs. Update README Musab Gültekin 2019-06-08 17:04:00 +03:00
  • edbddf74d8 Cache added to options Musab Gültekin 2019-06-08 16:01:31 +03:00
  • ca197ff06a Caching added. JSON File export will append, not truncate. Musab Gültekin 2019-06-08 15:29:09 +03:00
  • 9e61a96412 Sync and Async requests support added by using go keyword. Musab Gültekin 2019-06-07 21:31:18 +03:00
  • ee7d498f22 Allowed domains support added Musab Gültekin 2019-06-07 21:02:31 +03:00
  • 5271095c8e Export chan added. We can now export data by sending to this chan. Musab Gültekin 2019-06-07 20:52:31 +03:00
  • e58b08cbd6 Relative URL handling added to Response type. Musab Gültekin 2019-06-07 15:44:14 +03:00
  • 944bd3bada Options used instead of direct parameter passing. Tests updated. Musab Gültekin 2019-06-06 23:49:42 +03:00
  • 6358b87472 Use parse function to parse responses, instead of channels. Parse response as HTML Document using goquery. Added simple README. Musab Gültekin 2019-06-06 22:48:57 +03:00
  • 1c96048082 Initial commit Musab Gültekin 2019-06-06 17:11:19 +03:00