goscreenscraper

README

goscreenscraper

Go Screen Scrapper is library for screen scraping. It's support get and post method.

INSTALL

go get github.com/yogyrahmawan/goscreenscraper

USAGE

import (
	"fmt"
	"github.com/yogyrahmawan/goscreenscraper"
)

func main() {
  //example call without redirect, post method 
	m := make(map[string]string)
	m["log"] = "shyrin"
	m["pwd"] = "your_own_password"
	m["wp-submit"] = "Log In"
	m["redirect_to"] = "http://www.tourismjournal.net/wp-admin/"
	m["testcookie"] = "1"
	req2 := screenscraper.NewRequest("POST","http://www.tourismjournal.net","/wp-login.php","http://www.tourismjournal.net/wp-login.php?2yniv1rfyct10sdlt1d1r",m,nil)
	resp2,_ := req2.CallWithoutRedirect()

	fmt.Println(resp2.Cookies)

  //example call and redirect, get method
	req3 := screenscraper.NewRequest("GET","http://www.tourismjournal.net","/wp-admin/","",nil,resp2.Cookies)
	resp3,_ := req3.CallAndRedirect()

	fmt.Println(string(resp3.RawContent))
}

Release Notes

See Notes on release.txt

Author

Yogy Rahmawan

Documentation

Index

• Constants
• type HTTPRequest
  • func NewRequest(method string, baseUrl string, targetUrl string, referer string, ...) *HTTPRequest
  • func (httpRequest *HTTPRequest) CallAndRedirect() (*HttpReponse, error)
  • func (httpRequest *HTTPRequest) CallWithoutRedirect() (*HttpReponse, error)
• type HttpReponse
  • func NewResponse() *HttpReponse

Constants

const (
	METHOD_POST = "POST"
	METHOD_GET  = "GET"
)

constant for method post and get

Types

type HTTPRequest

type HTTPRequest struct {
	// contains filtered or unexported fields
}

*

• HTTP Request will be used for creating request
• method : method POST or GET
• postContents : contents that will be sent through request
• baseUrl : base url of site (ex : http://www.yoursite.com) . It must be start with http://
• referer : the previous url
• cookies : response cookies from previous page/login page

*

func NewRequest

func NewRequest(method string, baseUrl string, targetUrl string, referer string, formData map[string]string, cookies []*http.Cookie) *HTTPRequest

initialise request

func (*HTTPRequest) CallAndRedirect

func (httpRequest *HTTPRequest) CallAndRedirect() (*HttpReponse, error)

*

• CallAndRedirect use client for send request.
• It is implement cookieJar so if you want to send request using cookies, Use this method
• It is support redirect page *

func (*HTTPRequest) CallWithoutRedirect

func (httpRequest *HTTPRequest) CallWithoutRedirect() (*HttpReponse, error)

*

• CallWithoutRedirect method using roundtrip transport for sending data
• This method doesn't allow cookies when sending data
• It's also doesn't support redirect. Plese refer doc of http.Transport roundtrip
• I usually use this method to get cookie after successfull login *

type HttpReponse

type HttpReponse struct {
	RawContent []byte
	LastUrl    string
	Cookies    []*http.Cookie
}

*

• HTTP Response
• RawContent : array of byte from resp.Body
• lastUrl : landing page(redirect page)
• Cookies : cookies from response

*

func NewResponse

func NewResponse() *HttpReponse

initialise response