gitea/geziyor
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
128 Commits 1 Branch 0 Tags
a2a91b7b2eaaf056c1987c70f464fb8fd9b1af80
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Musab Gültekin a2a91b7b2e Default allocator options are used on rendered scraping. It can be changed using custom Client or changing client options after scraper creation. 2021-05-23 23:48:55 +03:00
Musab Gültekin 5aa2c2540e Default client function moved to client_test.go as its only used there. 2021-05-23 23:47:43 +03:00
Musab Gültekin f35d34bc02 chromedp library updated. 2021-05-23 23:14:47 +03:00
Musab Gültekin 16265e524d Response.JoinURL simplified. 2021-05-18 13:31:23 +03:00
Musab Gültekin 3c9a3849e2 Start command now waits for synchronized requests too. This fixes if requests are made using different goroutines with synchronized requests. 
It doesn't cause any issues on concurrent requests because we already wait for them.
 2021-04-19 12:58:47 +03:00
Musab Gültekin d28beca57a Fix race condition on hosts semaphore 2021-04-17 14:46:45 +03:00
Musab Gültekin c527d0b885 SIGINT (interrupt) signal receiving refactored and fixed working on some conditions 2021-04-17 14:11:17 +03:00
Musab Gültekin 6a23efd175 JoinURL now returns *url.URL and error 2021-04-17 11:12:22 +03:00
Musab Gültekin 9ea67b3554 Use fmt.Errorf instead of errors package. This is good convention after go 1.13 2021-04-17 11:11:29 +03:00
Musab Gültekin fbee722a38 Rate limiting per second implemented 2021-04-16 15:31:31 +03:00
Musab Gültekin d8252092f7 Add duplicate_requests_test.go 2021-04-16 14:43:42 +03:00
Musab Gültekin be4d13c0ef Retry checking refactored using util function. 2021-04-14 09:32:42 +03:00
Musab Gültekin 46c4db6b1a Exporters now need to return error. This is done because of simple error logging. 2021-04-14 09:30:17 +03:00
Musab Gültekin e3d79e2574 Added custom logger. Right now, not configurable. 2021-04-13 23:36:42 +03:00
Musab Gültekin 129402d754 Updated chromedp 2021-01-28 20:50:25 +03:00
Musab Gültekin 9b266b6cce Allocator options added 2021-01-28 20:49:01 +03:00
Musab Gültekin 29c29235ae Fixed response error if retrying disabled 2020-09-05 17:24:22 +03:00
Musab Gültekin 7a76a9b95e Allocators seperated for transparency. Updated chrome library. 2020-09-05 16:14:41 +03:00
Musab Gültekin cfb16fe1ee Call ErrorFunc on errors. Unexport DoRequestClient and DoRequestChrome 2019-12-13 00:03:44 +03:00
Musab Gültekin 7d2fe57bab Added error logging for HTML parser. 2019-12-11 13:55:38 +03:00
Musab Gültekin cbca22fefb Updated chrome protocol library 2019-11-16 20:34:57 +03:00
Musab Gültekin 6645820408 Added logging on allowed domains middleware and duplicate requests 2019-11-16 20:34:09 +03:00
Musab Gültekin 9b8a3837bd Added response joinURL test and updated chromedp. 2019-09-13 14:34:29 +03:00
Musab Gültekin 3264057679 Fixed issue on JoinURL 2019-08-06 17:21:41 +03:00
Musab Gültekin 86d4e80596 Added user-agent test, Fixed failing test 2019-08-05 16:18:44 +03:00
Musab Gültekin 85597219e6 Refactored client options 
Fixed default User-Agent string not being set.
 2019-08-05 15:42:30 +03:00
Musab Gültekin 0e5230eac8 Remote endpoint support added for js rendered requests. Geziyor is beta now. 2019-08-05 15:14:47 +03:00
Musab Gültekin c117d71fef Updated license 2019-08-05 15:01:48 +03:00
Musab Gültekin 32077d8433 Updated docs for rendered requests 2019-07-26 16:40:42 +03:00
Musab Gültekin e07ef4d66d Fixed important bug on rendering that was causing client request made too. Updated chromedp dependency 2019-07-26 16:07:09 +03:00
Musab Gültekin 762854e511 Go 1.10 and 1.11 support added by using different methods on reflect package. 2019-07-21 12:08:41 +03:00
Musab Gültekin df37629d4d Disabled indenting on JSON exporter as it looks so ugly on exported data. 
JSONLine still supports indenting.
 2019-07-14 03:37:52 +03:00
Musab Gültekin dfabcb84fd JSON renamed to JSONLine. JSON List support added. 2019-07-14 03:30:59 +03:00
Musab Gültekin d19465c44a Robotstxt metrics added. 2019-07-08 14:51:54 +03:00
Musab Gültekin d3c4389c46 Retrying support added for chrome. Fixed robots.txt retry issue. Fixed Meta issue 2019-07-07 19:50:15 +03:00
Musab Gültekin 90d2be2210 Caching policies added. 
We used httpcache library to implement this. As it was not possible to support different policies, I mostly copied and modified it.
 2019-07-07 12:18:40 +03:00
Musab Gültekin 0d6c2a6864 Graceful shut down system implemented 2019-07-06 18:32:13 +03:00
Musab Gültekin 42faa92ece Robots.txt support implemented 2019-07-06 16:18:03 +03:00
Musab Gültekin 2cab68d2ce Middlewares refactored to multiple files in middleware package. 
Extractors removed as they introduce complexity to scraper. Both in learning and developing.
 2019-07-04 21:04:29 +03:00
Musab Gültekin 9adff75509 Retry requests support implemented for client. 2019-07-04 13:36:10 +03:00
Musab Gültekin da03567fae Extractors refactored to support pass by value. Documentation added for request and response. 2019-07-04 02:13:29 +03:00
Musab Gültekin 71683ec6de Chardet removed as its not good enough to detect. Built-int library is good enough. 2019-07-03 20:54:17 +03:00
Musab Gültekin 33238bc875 Charset detection heuristics added with chardet lib. 2019-07-03 18:08:28 +03:00
Musab Gültekin b355a566cf Added more tests and refactored exporter tests. Added code coverage badge. 2019-07-02 14:53:06 +03:00
Musab Gültekin 4ab7cfd904 Exporter and Extractor interfaces moved to its own package for simplicity of main Geziyor package 2019-07-02 13:22:23 +03:00
Musab Gültekin c0dd0393e6 Maximum redirection option added. Performance improvement on exports. Duplicate requests only checked on GET requests. 2019-07-01 15:44:28 +03:00
Musab Gültekin 80f3500a69 Fixed Chrome response not right on some sites. 2019-07-01 12:32:15 +03:00
Musab Gültekin fb5b4e3406 README updated according to new package names 2019-06-30 22:21:36 +03:00
Musab Gültekin 0eda056065 Attribute extractor added. HTML extractor added. Outer HTML Extractor added. 
exporter package renamed to export, extractor package renamed to extract for simplicity.
 2019-06-30 22:20:17 +03:00
Musab Gültekin 7c383b175f Metrics Server support added for expvar. Refactored some methods. 2019-06-30 19:09:03 +03:00
Musab Gültekin ec4551a8a0 Making Requests and reading responses refactored to client package. 2019-06-30 16:21:18 +03:00
Musab Gültekin 0eac5f5f40 Fixed exporters bug that was causing last exported items not written to disk. 2019-06-29 16:11:52 +03:00
Musab Gültekin bd6466a5f2 http package renamed to client to reduce cunfusion 2019-06-29 14:18:31 +03:00
Musab Gültekin 1e109c555d Request and response moved to http package 2019-06-29 13:36:39 +03:00
Musab Gültekin 59757607eb Pretty print exporter added. Panic counter added to metrics 2019-06-29 11:20:06 +03:00
Musab Gültekin 276b248ebb Synchronized requests support added. Benchmarks added. 2019-06-28 17:28:16 +03:00
Musab Gültekin b000581c3d Extractors implemented. Exporters name simplified. README Updated for extracting data. Removed go 1.11 support 2019-06-28 13:00:30 +03:00
Musab Gültekin 679fd8ab7a Map support added for CSV exporter 2019-06-27 22:39:06 +03:00
Musab Gültekin 8fe194bd10 Added options and tests for exporters. 2019-06-27 16:54:09 +03:00
Musab Gültekin d20ea47390 Fix Header convertion bug. Map was not canonicalizing keys 2019-06-22 15:04:08 +03:00
Musab Gültekin 02df5aa4e8 Fixed issues on non-trailing URLS on rendered requests 2019-06-22 14:47:12 +03:00
Musab Gültekin 92e7cfefec Fixed README Doc. 2019-06-22 13:13:33 +03:00
Musab Gültekin a64a262554 HTTP Client can be changed now. Docs updated. 2019-06-22 13:12:05 +03:00
Musab Gültekin 7bc782400c Expvar metrics support added. Metrics refactored to its own package. 2019-06-21 21:37:25 +03:00
Musab Gültekin 88c4b1dd35 Prometheus metrics support added. 2019-06-21 20:05:28 +03:00
Musab Gültekin 141bab0d05 Error handling improved 2019-06-20 10:14:36 +03:00
Musab Gültekin f88b88986c Delays and logs refactored as middlewares. 2019-06-20 09:54:30 +03:00
Musab Gültekin 514fe2e8d2 Recover system refactored like middleware 2019-06-19 22:45:40 +03:00
Musab Gültekin c28b228a12 Response header bug fixed for Chrome 2019-06-18 16:37:06 +03:00
Musab Gültekin ec83a92eb3 Response header support added for Chrome Rendering 2019-06-18 16:26:40 +03:00
Musab Gültekin 217f3c96df Header and native http.Response support added for Chrome rendering 2019-06-18 16:16:29 +03:00
Musab Gültekin 936d157785 Revert "Try parsing HTML even if content-type is empty." 
This reverts commit f384fc2c
 2019-06-18 13:03:00 +03:00
Musab Gültekin f384fc2c13 Try parsing HTML even if content-type is empty. 2019-06-18 13:00:16 +03:00
Musab Gültekin 4177f10de9 Request creation simplified and basic auth test added. 2019-06-17 13:53:34 +03:00
Musab Gültekin a5ec28664d Cookies support added. 2019-06-17 13:31:19 +03:00
Musab Gültekin dd6687f976 Fixed build issue 2019-06-17 12:21:40 +03:00
Musab Gültekin e50fa3b1dc Response middlewares support implemented. 2019-06-16 18:29:07 +03:00
Musab Gültekin 80383ebd6f Middlewares and some string util functions refactored. Added partial Documentation. 2019-06-16 10:38:03 +03:00
Musab Gültekin 40f673f2e2 Fixed README. More Go versions added for testing 2019-06-15 22:35:51 +03:00
Musab Gültekin ddff3aee25 Request cancellations support added to Middlewares. 
Some core functions refactored as middlewares.
Fixed race condition in exporting system. Now, only one goroutine will be responsible for exporting. This fixes concurrency issues on writing.
 2019-06-15 22:27:46 +03:00
Musab Gültekin 7b23596a2d Middleware support added. HTML Parsing disable option added. 
Goroutine leaks will be tested using leaktest lib.
 2019-06-15 17:55:40 +03:00
Musab Gültekin 4799b0f7b4 Fixed goroutine leaks. Updated travis build 2019-06-14 17:30:49 +03:00
Musab Gültekin f5b3b0d049 Fixed race conditions on exporters. 
MaxIdleConns limit disabled to support unlimited requests to all hosts.
MaxIdleConnsPerHost limit increased to speed up requests to same host.
 2019-06-14 16:10:36 +03:00
Musab Gültekin c6ea82447a Travis-CI configuration added. Now, we'll test coverage on every commit. 2019-06-14 15:24:15 +03:00
Musab Gültekin 6caf1effd6 Rendered field exported to support rendered requests on Do function. Data races fixed. 2019-06-14 15:23:56 +03:00
Musab Gültekin 1a7d480b36 JS Rendered requests with Chrome support added 2019-06-13 22:08:45 +03:00
Musab Gültekin 76a687e193 Do function refactored 2019-06-13 20:26:07 +03:00
Musab Gültekin 8a6e19a031 New requests on StartRequests func will be made using Geziyor's methods. Not Requests chan 
Options field exported.
 2019-06-13 14:06:37 +03:00
Musab Gültekin 184081d3bf README updated for more advanced usage. Updated tests. 2019-06-12 22:22:01 +03:00
Musab Gültekin d56ea161a5 Making new requests on StartRequestsFunc is simplified by using channels 2019-06-12 21:54:57 +03:00
Musab Gültekin f7f4e401e2 Metadata adding on requests support added. StartRequests function implemented. 2019-06-12 21:30:45 +03:00
Musab Gültekin bd8d58576f Start requests function implemented. 2019-06-12 12:40:38 +03:00
Musab Gültekin 2f6cb06982 Disabling charset detection implemented. 2019-06-12 11:44:31 +03:00
Musab Gültekin a311a0f998 CSV exporter support added. Not finished for map type. 2019-06-11 20:42:22 +03:00
Musab Gültekin bbdc3bcacd Exporters made optional, as some scrapers only want to see data in console. 2019-06-11 18:59:37 +03:00
Musab Gültekin b8305d5e1a Limiting body reading support implemented. 2019-06-11 16:19:30 +03:00
Musab Gültekin 3790295658 Multiple Exporters and custom Exporters support added. 2019-06-11 16:10:49 +03:00
Musab Gültekin e4e8723426 Callback are now mandatory as almost all the scrapers use it. 2019-06-11 14:24:48 +03:00
Musab Gültekin ca2414c5c8 Request callbacks added. 
Recover from all panics and continue scraping.
Only parse HTML if response is HTML.
 2019-06-09 21:13:30 +03:00
Musab Gültekin 7abc7a370d Disabling logs support added. 2019-06-09 19:14:46 +03:00