Musab Gültekin
c0dd0393e6
Maximum redirection option added. Performance improvement on exports. Duplicate requests only checked on GET requests.
2019-07-01 15:44:28 +03:00
Musab Gültekin
80f3500a69
Fixed Chrome response not right on some sites.
2019-07-01 12:32:15 +03:00
Musab Gültekin
0eda056065
Attribute extractor added. HTML extractor added. Outer HTML Extractor added.
...
exporter package renamed to export, extractor package renamed to extract for simplicity.
2019-06-30 22:20:17 +03:00
Musab Gültekin
7c383b175f
Metrics Server support added for expvar. Refactored some methods.
2019-06-30 19:09:03 +03:00
Musab Gültekin
ec4551a8a0
Making Requests and reading responses refactored to client package.
2019-06-30 16:21:18 +03:00
Musab Gültekin
0eac5f5f40
Fixed exporters bug that was causing last exported items not written to disk.
2019-06-29 16:11:52 +03:00
Musab Gültekin
bd6466a5f2
http package renamed to client to reduce cunfusion
2019-06-29 14:18:31 +03:00
Musab Gültekin
1e109c555d
Request and response moved to http package
2019-06-29 13:36:39 +03:00
Musab Gültekin
59757607eb
Pretty print exporter added. Panic counter added to metrics
2019-06-29 11:20:06 +03:00
Musab Gültekin
276b248ebb
Synchronized requests support added. Benchmarks added.
2019-06-28 17:28:16 +03:00
Musab Gültekin
b000581c3d
Extractors implemented. Exporters name simplified. README Updated for extracting data. Removed go 1.11 support
2019-06-28 13:00:30 +03:00
Musab Gültekin
02df5aa4e8
Fixed issues on non-trailing URLS on rendered requests
2019-06-22 14:47:12 +03:00
Musab Gültekin
a64a262554
HTTP Client can be changed now. Docs updated.
2019-06-22 13:12:05 +03:00
Musab Gültekin
7bc782400c
Expvar metrics support added. Metrics refactored to its own package.
2019-06-21 21:37:25 +03:00
Musab Gültekin
88c4b1dd35
Prometheus metrics support added.
2019-06-21 20:05:28 +03:00
Musab Gültekin
141bab0d05
Error handling improved
2019-06-20 10:14:36 +03:00
Musab Gültekin
f88b88986c
Delays and logs refactored as middlewares.
2019-06-20 09:54:30 +03:00
Musab Gültekin
514fe2e8d2
Recover system refactored like middleware
2019-06-19 22:45:40 +03:00
Musab Gültekin
c28b228a12
Response header bug fixed for Chrome
2019-06-18 16:37:06 +03:00
Musab Gültekin
ec83a92eb3
Response header support added for Chrome Rendering
2019-06-18 16:26:40 +03:00
Musab Gültekin
217f3c96df
Header and native http.Response support added for Chrome rendering
2019-06-18 16:16:29 +03:00
Musab Gültekin
4177f10de9
Request creation simplified and basic auth test added.
2019-06-17 13:53:34 +03:00
Musab Gültekin
a5ec28664d
Cookies support added.
2019-06-17 13:31:19 +03:00
Musab Gültekin
e50fa3b1dc
Response middlewares support implemented.
2019-06-16 18:29:07 +03:00
Musab Gültekin
80383ebd6f
Middlewares and some string util functions refactored. Added partial Documentation.
2019-06-16 10:38:03 +03:00
Musab Gültekin
ddff3aee25
Request cancellations support added to Middlewares.
...
Some core functions refactored as middlewares.
Fixed race condition in exporting system. Now, only one goroutine will be responsible for exporting. This fixes concurrency issues on writing.
2019-06-15 22:27:46 +03:00
Musab Gültekin
7b23596a2d
Middleware support added. HTML Parsing disable option added.
...
Goroutine leaks will be tested using leaktest lib.
2019-06-15 17:55:40 +03:00
Musab Gültekin
4799b0f7b4
Fixed goroutine leaks. Updated travis build
2019-06-14 17:30:49 +03:00
Musab Gültekin
f5b3b0d049
Fixed race conditions on exporters.
...
MaxIdleConns limit disabled to support unlimited requests to all hosts.
MaxIdleConnsPerHost limit increased to speed up requests to same host.
2019-06-14 16:10:36 +03:00
Musab Gültekin
6caf1effd6
Rendered field exported to support rendered requests on Do function. Data races fixed.
2019-06-14 15:23:56 +03:00
Musab Gültekin
1a7d480b36
JS Rendered requests with Chrome support added
2019-06-13 22:08:45 +03:00
Musab Gültekin
76a687e193
Do function refactored
2019-06-13 20:26:07 +03:00
Musab Gültekin
8a6e19a031
New requests on StartRequests func will be made using Geziyor's methods. Not Requests chan
...
Options field exported.
2019-06-13 14:06:37 +03:00
Musab Gültekin
d56ea161a5
Making new requests on StartRequestsFunc is simplified by using channels
2019-06-12 21:54:57 +03:00
Musab Gültekin
f7f4e401e2
Metadata adding on requests support added. StartRequests function implemented.
2019-06-12 21:30:45 +03:00
Musab Gültekin
bd8d58576f
Start requests function implemented.
2019-06-12 12:40:38 +03:00
Musab Gültekin
2f6cb06982
Disabling charset detection implemented.
2019-06-12 11:44:31 +03:00
Musab Gültekin
a311a0f998
CSV exporter support added. Not finished for map type.
2019-06-11 20:42:22 +03:00
Musab Gültekin
bbdc3bcacd
Exporters made optional, as some scrapers only want to see data in console.
2019-06-11 18:59:37 +03:00
Musab Gültekin
b8305d5e1a
Limiting body reading support implemented.
2019-06-11 16:19:30 +03:00
Musab Gültekin
3790295658
Multiple Exporters and custom Exporters support added.
2019-06-11 16:10:49 +03:00
Musab Gültekin
e4e8723426
Callback are now mandatory as almost all the scrapers use it.
2019-06-11 14:24:48 +03:00
Musab Gültekin
ca2414c5c8
Request callbacks added.
...
Recover from all panics and continue scraping.
Only parse HTML if response is HTML.
2019-06-09 21:13:30 +03:00
Musab Gültekin
7abc7a370d
Disabling logs support added.
2019-06-09 19:14:46 +03:00
Musab Gültekin
b973c1c064
Request delays support added
2019-06-09 14:24:53 +03:00
Musab Gültekin
2263108838
User-Agent change support added.
2019-06-09 13:43:17 +03:00
Musab Gültekin
9263877339
Exporting data all types support added.
2019-06-09 13:22:20 +03:00
Musab Gültekin
d967555b62
Global and Domain Concurrency limit implemented. Updated README
2019-06-09 11:53:40 +03:00
Musab Gültekin
a9aaf86df3
Automatic determining response and decoding it.
2019-06-09 10:46:32 +03:00
Musab Gültekin
f2fba85266
Timeout support added
2019-06-08 20:36:43 +03:00