dd6687f976
Fixed build issue
2019-06-17 12:21:40 +03:00
e50fa3b1dc
Response middlewares support implemented.
2019-06-16 18:29:07 +03:00
80383ebd6f
Middlewares and some string util functions refactored. Added partial Documentation.
2019-06-16 10:38:03 +03:00
40f673f2e2
Fixed README. More Go versions added for testing
2019-06-15 22:35:51 +03:00
ddff3aee25
Request cancellations support added to Middlewares.
...
Some core functions refactored as middlewares.
Fixed race condition in exporting system. Now, only one goroutine will be responsible for exporting. This fixes concurrency issues on writing.
2019-06-15 22:27:46 +03:00
83a7b9eb87
Merge pull request #4 from NMelis/master
...
Create CONTRIBUTING.md
2019-06-15 18:09:58 +03:00
f65456f18c
Update CONTRIBUTING.md
2019-06-15 18:08:27 +03:00
7b23596a2d
Middleware support added. HTML Parsing disable option added.
...
Goroutine leaks will be tested using leaktest lib.
2019-06-15 17:55:40 +03:00
2e29c47acd
Create CONTRIBUTING.md
...
could add a description of how to become a contributor?
2019-06-15 19:43:49 +07:00
4799b0f7b4
Fixed goroutine leaks. Updated travis build
2019-06-14 17:30:49 +03:00
f5b3b0d049
Fixed race conditions on exporters.
...
MaxIdleConns limit disabled to support unlimited requests to all hosts.
MaxIdleConnsPerHost limit increased to speed up requests to same host.
2019-06-14 16:10:36 +03:00
83bfb01856
Merge pull request #3 from isacikgoz/master
...
Update README.md
2019-06-14 15:34:31 +03:00
b2f32b8830
Merge branch 'master' into master
2019-06-14 15:32:36 +03:00
c6ea82447a
Travis-CI configuration added. Now, we'll test coverage on every commit.
2019-06-14 15:24:15 +03:00
6caf1effd6
Rendered field exported to support rendered requests on Do function. Data races fixed.
2019-06-14 15:23:56 +03:00
7360ffa3c9
Update README.md
2019-06-14 14:57:53 +03:00
1a7d480b36
JS Rendered requests with Chrome support added
2019-06-13 22:08:45 +03:00
76a687e193
Do function refactored
2019-06-13 20:26:07 +03:00
7c43585769
Fixed license copyright
2019-06-13 15:48:02 +03:00
8a6e19a031
New requests on StartRequests func will be made using Geziyor's methods. Not Requests chan
...
Options field exported.
2019-06-13 14:06:37 +03:00
184081d3bf
README updated for more advanced usage. Updated tests.
2019-06-12 22:22:01 +03:00
d56ea161a5
Making new requests on StartRequestsFunc is simplified by using channels
2019-06-12 21:54:57 +03:00
f7f4e401e2
Metadata adding on requests support added. StartRequests function implemented.
2019-06-12 21:30:45 +03:00
bd8d58576f
Start requests function implemented.
2019-06-12 12:40:38 +03:00
2f6cb06982
Disabling charset detection implemented.
2019-06-12 11:44:31 +03:00
a311a0f998
CSV exporter support added. Not finished for map type.
2019-06-11 20:42:22 +03:00
bbdc3bcacd
Exporters made optional, as some scrapers only want to see data in console.
2019-06-11 18:59:37 +03:00
b8305d5e1a
Limiting body reading support implemented.
2019-06-11 16:19:30 +03:00
3790295658
Multiple Exporters and custom Exporters support added.
2019-06-11 16:10:49 +03:00
e4e8723426
Callback are now mandatory as almost all the scrapers use it.
2019-06-11 14:24:48 +03:00
ca2414c5c8
Request callbacks added.
...
Recover from all panics and continue scraping.
Only parse HTML if response is HTML.
2019-06-09 21:13:30 +03:00
7abc7a370d
Disabling logs support added.
2019-06-09 19:14:46 +03:00
b973c1c064
Request delays support added
2019-06-09 14:24:53 +03:00
2263108838
User-Agent change support added.
2019-06-09 13:43:17 +03:00
9263877339
Exporting data all types support added.
2019-06-09 13:22:20 +03:00
d967555b62
Global and Domain Concurrency limit implemented. Updated README
2019-06-09 11:53:40 +03:00
a9aaf86df3
Automatic determining response and decoding it.
2019-06-09 10:46:32 +03:00
f2fba85266
Timeout support added
2019-06-08 20:36:43 +03:00
2e3bd18430
Options refactored to its own file. Timeout increased to 60 sec
2019-06-08 20:36:43 +03:00
b90908066b
Head API added. Opt renamed to Options. Tests updated. More documentation added.
2019-06-08 20:36:43 +03:00
95d97436bf
Create LICENSE.txt
2019-06-08 19:59:49 +03:00
815ae7eec5
Do request support added. Updated docs.
2019-06-08 19:45:48 +03:00
54c7d3550f
Gezer renamed to Geziyor
2019-06-08 17:14:10 +03:00
c525e0d7d0
Don't visit already visited URLs. Update README
2019-06-08 17:04:00 +03:00
edbddf74d8
Cache added to options
2019-06-08 16:01:31 +03:00
ca197ff06a
Caching added.
...
JSON File export will append, not truncate.
2019-06-08 15:29:09 +03:00
9e61a96412
Sync and Async requests support added by using go keyword.
2019-06-07 21:31:18 +03:00
ee7d498f22
Allowed domains support added
2019-06-07 21:02:31 +03:00
5271095c8e
Export chan added. We can now export data by sending to this chan.
2019-06-07 20:52:31 +03:00
e58b08cbd6
Relative URL handling added to Response type.
2019-06-07 15:44:14 +03:00
