Musab Gültekin
|
184081d3bf
|
README updated for more advanced usage. Updated tests.
|
2019-06-12 22:22:01 +03:00 |
|
Musab Gültekin
|
d56ea161a5
|
Making new requests on StartRequestsFunc is simplified by using channels
|
2019-06-12 21:54:57 +03:00 |
|
Musab Gültekin
|
f7f4e401e2
|
Metadata adding on requests support added. StartRequests function implemented.
|
2019-06-12 21:30:45 +03:00 |
|
Musab Gültekin
|
bd8d58576f
|
Start requests function implemented.
|
2019-06-12 12:40:38 +03:00 |
|
Musab Gültekin
|
2f6cb06982
|
Disabling charset detection implemented.
|
2019-06-12 11:44:31 +03:00 |
|
Musab Gültekin
|
a311a0f998
|
CSV exporter support added. Not finished for map type.
|
2019-06-11 20:42:22 +03:00 |
|
Musab Gültekin
|
bbdc3bcacd
|
Exporters made optional, as some scrapers only want to see data in console.
|
2019-06-11 18:59:37 +03:00 |
|
Musab Gültekin
|
b8305d5e1a
|
Limiting body reading support implemented.
|
2019-06-11 16:19:30 +03:00 |
|
Musab Gültekin
|
3790295658
|
Multiple Exporters and custom Exporters support added.
|
2019-06-11 16:10:49 +03:00 |
|
Musab Gültekin
|
e4e8723426
|
Callback are now mandatory as almost all the scrapers use it.
|
2019-06-11 14:24:48 +03:00 |
|
Musab Gültekin
|
ca2414c5c8
|
Request callbacks added.
Recover from all panics and continue scraping.
Only parse HTML if response is HTML.
|
2019-06-09 21:13:30 +03:00 |
|
Musab Gültekin
|
7abc7a370d
|
Disabling logs support added.
|
2019-06-09 19:14:46 +03:00 |
|
Musab Gültekin
|
b973c1c064
|
Request delays support added
|
2019-06-09 14:24:53 +03:00 |
|
Musab Gültekin
|
2263108838
|
User-Agent change support added.
|
2019-06-09 13:43:17 +03:00 |
|
Musab Gültekin
|
9263877339
|
Exporting data all types support added.
|
2019-06-09 13:22:20 +03:00 |
|
Musab Gültekin
|
d967555b62
|
Global and Domain Concurrency limit implemented. Updated README
|
2019-06-09 11:53:40 +03:00 |
|
Musab Gültekin
|
a9aaf86df3
|
Automatic determining response and decoding it.
|
2019-06-09 10:46:32 +03:00 |
|
Musab Gültekin
|
f2fba85266
|
Timeout support added
|
2019-06-08 20:36:43 +03:00 |
|
Musab Gültekin
|
2e3bd18430
|
Options refactored to its own file. Timeout increased to 60 sec
|
2019-06-08 20:36:43 +03:00 |
|
Musab Gültekin
|
b90908066b
|
Head API added. Opt renamed to Options. Tests updated. More documentation added.
|
2019-06-08 20:36:43 +03:00 |
|
Musab Gültekin
|
95d97436bf
|
Create LICENSE.txt
|
2019-06-08 19:59:49 +03:00 |
|
Musab Gültekin
|
815ae7eec5
|
Do request support added. Updated docs.
|
2019-06-08 19:45:48 +03:00 |
|
Musab Gültekin
|
54c7d3550f
|
Gezer renamed to Geziyor
|
2019-06-08 17:14:10 +03:00 |
|
Musab Gültekin
|
c525e0d7d0
|
Don't visit already visited URLs. Update README
|
2019-06-08 17:04:00 +03:00 |
|
Musab Gültekin
|
edbddf74d8
|
Cache added to options
|
2019-06-08 16:01:31 +03:00 |
|
Musab Gültekin
|
ca197ff06a
|
Caching added.
JSON File export will append, not truncate.
|
2019-06-08 15:29:09 +03:00 |
|
Musab Gültekin
|
9e61a96412
|
Sync and Async requests support added by using go keyword.
|
2019-06-07 21:31:18 +03:00 |
|
Musab Gültekin
|
ee7d498f22
|
Allowed domains support added
|
2019-06-07 21:02:31 +03:00 |
|
Musab Gültekin
|
5271095c8e
|
Export chan added. We can now export data by sending to this chan.
|
2019-06-07 20:52:31 +03:00 |
|
Musab Gültekin
|
e58b08cbd6
|
Relative URL handling added to Response type.
|
2019-06-07 15:44:14 +03:00 |
|
Musab Gültekin
|
944bd3bada
|
Options used instead of direct parameter passing.
Tests updated.
|
2019-06-06 23:49:42 +03:00 |
|
Musab Gültekin
|
6358b87472
|
Use parse function to parse responses, instead of channels.
Parse response as HTML Document using goquery.
Added simple README.
|
2019-06-06 22:48:57 +03:00 |
|
Musab Gültekin
|
1c96048082
|
Initial commit
|
2019-06-06 17:11:19 +03:00 |
|