gitea/geziyor
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Update README.md

Browse Source
This commit is contained in:
Ibrahim Serdar Acikgoz 2019-06-14 14:57:53 +03:00 committed by GitHub
parent 1a7d480b36
commit 7360ffa3c9
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 17 additions and 17 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

34
README.md
View File

@ -1,5 +1,5 @@
# Geziyor # Geziyor
Geziyor is a blazing fast web crawling and web scraping framework, used to crawl websites and extract structured data from their pages. It can be used for a wide range of purposes, from data mining to monitoring and automated testing. Geziyor is a blazing fast web crawling and web scraping framework. It can be used to crawl websites and extract structured data from them. Geziyor is useful for a wide range of purposes such as data mining, monitoring and automated testing.
[![GoDoc](https://godoc.org/github.com/geziyor/geziyor?status.svg)](https://godoc.org/github.com/geziyor/geziyor) [![GoDoc](https://godoc.org/github.com/geziyor/geziyor?status.svg)](https://godoc.org/github.com/geziyor/geziyor)
[![report card](https://goreportcard.com/badge/github.com/geziyor/geziyor)](http://goreportcard.com/report/geziyor/geziyor) [![report card](https://goreportcard.com/badge/github.com/geziyor/geziyor)](http://goreportcard.com/report/geziyor/geziyor)
@ -16,7 +16,7 @@ Geziyor is a blazing fast web crawling and web scraping framework, used to crawl
See scraper [Options](https://godoc.org/github.com/geziyor/geziyor#Options) for all custom settings. See scraper [Options](https://godoc.org/github.com/geziyor/geziyor#Options) for all custom settings.
## Status ## Status
We highly recommend you to use go modules. As this project is in **development stage** right now and **API is not stable**. Since the project is in **development phase**, **API may change in time**. Also, we highly recommend you to use Geziyor with go modules.
## Usage ## Usage
Simple usage Simple usage
@ -34,27 +34,27 @@ Advanced usage
```go ```go
func main() { func main() {
geziyor.NewGeziyor(geziyor.Options{ geziyor.NewGeziyor(geziyor.Options{
StartURLs: []string{"http://quotes.toscrape.com/"}, StartURLs: []string{"http://quotes.toscrape.com/"},
ParseFunc: quotesParse, ParseFunc: quotesParse,
Exporters: []geziyor.Exporter{exporter.JSONExporter{}}, Exporters: []geziyor.Exporter{exporter.JSONExporter{}},
}).Start() }).Start()
} }
func quotesParse(r *geziyor.Response) { func quotesParse(r *geziyor.Response) {
r.DocHTML.Find("div.quote").Each(func(i int, s *goquery.Selection) { r.DocHTML.Find("div.quote").Each(func(i int, s *goquery.Selection) {
r.Exports <- map[string]interface{}{ r.Exports <- map[string]interface{}{
"text": s.Find("span.text").Text(), "text": s.Find("span.text").Text(),
"author": s.Find("small.author").Text(), "author": s.Find("small.author").Text(),
} }
}) })
if href, ok := r.DocHTML.Find("li.next > a").Attr("href"); ok { if href, ok := r.DocHTML.Find("li.next > a").Attr("href"); ok {
go r.Geziyor.Get(r.JoinURL(href), quotesParse) go r.Geziyor.Get(r.JoinURL(href), quotesParse)
} }
} }
``` ```
See [tests](https://github.com/geziyor/geziyor/blob/master/geziyor_test.go) for more usage examples See [tests](https://github.com/geziyor/geziyor/blob/master/geziyor_test.go) for more usage examples.
## Installation ## Installation