Go: Building and Using a Simple Web Scraper

Prerequisites

Before we begin, ensure you have the following:

• Go installed on your machine. You can download it from golang.org.
• A basic understanding of Go syntax and concepts.

Step 1: Setting Up Your Go Environment

First, create a new directory for your project and navigate to it:

mkdir go-web-scraper
cd go-web-scraper

Next, initialize a new Go module:

go mod init go-web-scraper

Step 2: Installing the Colly Library

To install the colly package, run the following command:

go get -u github.com/gocolly/colly/v2

This command fetches the colly library and adds it to your project’s dependencies.

Step 3: Writing the Scraper

Create a new file named main.go and open it in your favorite text editor. Add the following code:

package main

import (
    "fmt"
    "log"

    "github.com/gocolly/colly/v2"
)

func main() {
    // Create a new collector
    c := colly.NewCollector()

    // Set up a callback for when a visited HTML element is found
    c.OnHTML("h1", func(e *colly.HTMLElement) {
        fmt.Println("Found H1:", e.Text)
    })

    // Set up a callback for when a visited HTML element is found
    c.OnHTML("a[href]", func(e *colly.HTMLElement) {
        link := e.Attr("href")
        fmt.Println("Found link:", link)
    })

    // Start the web scraping by visiting the target URL
    err := c.Visit("https://example.com")
    if err != nil {
        log.Fatal(err)
    }
}

Step 4: Running the Scraper

To run your web scraper, execute the following command in your terminal:

go run main.go

You should see output similar to:

Found H1: Example Domain
Found link: https://www.iana.org/domains/example

Step 5: Understanding the Code

• Collector: The colly.NewCollector() function creates a new collector instance, which is responsible for managing the scraping process.
• Callbacks: The OnHTML method allows you to define callbacks for specific HTML elements. In this case, we are looking for <h1> tags and anchor (<a>) tags with href attributes.
• Visiting URLs: The Visit method initiates the scraping process by requesting the specified URL.

Step 6: Handling Errors and Rate Limiting

To make your scraper more robust, you should handle errors and implement rate limiting to avoid overwhelming the target server. Here’s how to do that:

c.OnError(func(r *colly.Response, err error) {
    log.Println("Request failed:", r.Request.URL, err)
})

// Set a delay between requests to avoid hitting the server too hard
c.Limit(&colly.Limit{
    Delay: 2 * time.Second,
})

Step 7: Storing Scraped Data

You may want to store the scraped data for further analysis. Here’s an example of how to save the results to a CSV file:

import (
    "encoding/csv"
    "os"
)

// Create a CSV file
file, err := os.Create("scraped_data.csv")
if err != nil {
    log.Fatal(err)
}
defer file.Close()

writer := csv.NewWriter(file)
defer writer.Flush()

// Write header
writer.Write([]string{"Title", "Link"})

// Modify the existing callbacks to write data to CSV
c.OnHTML("h1", func(e *colly.HTMLElement) {
    writer.Write([]string{e.Text, ""}) // Placeholder for link
})

c.OnHTML("a[href]", func(e *colly.HTMLElement) {
    writer.Write([]string{"", e.Attr("href")}) // Placeholder for title
})

Best Practices for Web Scraping

Conclusion

In this tutorial, you learned how to build a simple web scraper using Go and the colly library. We covered the setup process, writing the scraper, handling errors, and storing data. By following best practices, you can ensure that your web scraping activities are effective and respectful.

Learn more with useful resources

• Colly Documentation
• Web Scraping with Go
• Go by Example: Web Scraping