Geziyor is a blazing fast web crawling and web scraping framework, used to crawl websites and extract structured data from their pages. It can be used for a wide range of purposes, from data mining to monitoring and automated testing.

Features

1.000+ Requests/Sec
Caching (Memory/Disk)
Automatic Data Exporting
Limit Concurrency (Global/Per Domain)
Request Delays (Constant/Randomized)
Automatic response decoding to UTF-8

See scraper Options for customization.

Usage

Simplest usage

geziyor.NewGeziyor(geziyor.Options{
    StartURLs: []string{"http://api.ipify.org"},
    ParseFunc: func(r *geziyor.Response) {
        fmt.Println(string(r.Body))
    },
}).Start()

Status

We highly recommend you to use go modules. As this project is in development stage right now and API is not stable.

Installation

go get github.com/geziyor/geziyor