Geziyor

Geziyor is a blazing fast web crawling and web scraping framework, used to crawl websites and extract structured data from their pages. It can be used for a wide range of purposes, from data mining to monitoring and automated testing.

Features

• 1.000+ Requests/Sec
• Caching (Memory/Disk)
• Automatic Data Exporting (JSON, CSV, or custom)
• Limit Concurrency (Global/Per Domain)
• Request Delays (Constant/Randomized)
• Automatic response decoding to UTF-8

See scraper Options for customization.

Status

We highly recommend you to use go modules. As this project is in development stage right now and API is not stable.

Usage

Simplest usage

geziyor.NewGeziyor(geziyor.Options{
    StartURLs: []string{"http://api.ipify.org"},
    ParseFunc: func(r *geziyor.Response) {
        fmt.Println(string(r.Body))
    },
}).Start()

Installation

go get github.com/geziyor/geziyor