Documentation
¶
Overview ¶
Package job includes mainly 2 interface: Processor & Collector.
Processor is supposed to receive a single string (refered to as a "msg") as input, then it outputs one or more semi-finished results(refered to as a "item") via channels to Collector. Collector receive the items and flush them out when appropriate or do anything you like.
Index ¶
- Constants
- func BKDRHash(str string) int
- func FilePutContentAppend(file string, content string) error
- func GetLeastBusyChannel(channels [](chan Item)) int
- func GetShortHostName() (string, error)
- type Collector
- type DocIdInfo
- type Item
- type LogInfo
- type LogtoHdfsCollector
- type LogtoHdfsProcessor
- type MessageQueue
- type Processor
- type SLMap
- type StrSlice
- type TestCollector
- type TestProcessor
- type TimeSlice
Constants ¶
View Source
const TimeFormatHour = "2006-01-02-15"
View Source
const TimeFormatMinute = "2006-01-02-15-04"
Variables ¶
This section is empty.
Functions ¶
func FilePutContentAppend ¶
func GetLeastBusyChannel ¶
得到len()最小的channel,做负载均衡使用
func GetShortHostName ¶
Types ¶
type Collector ¶
type Collector interface { Init(ctx *sj.Json, id int) error Collect(item Item) error Tick() error Destory() error }
func NewCollector ¶
type LogtoHdfsCollector ¶
type LogtoHdfsCollector struct {
// contains filtered or unexported fields
}
func (*LogtoHdfsCollector) Collect ¶
func (p *LogtoHdfsCollector) Collect(item Item) error
func (*LogtoHdfsCollector) Destory ¶
func (p *LogtoHdfsCollector) Destory() error
func (*LogtoHdfsCollector) HandleZip ¶
func (p *LogtoHdfsCollector) HandleZip(path string) (string, error)
func (*LogtoHdfsCollector) Tick ¶
func (p *LogtoHdfsCollector) Tick() error
type LogtoHdfsProcessor ¶
type LogtoHdfsProcessor struct {
// contains filtered or unexported fields
}
receive local paths, and logto hadoop
func (*LogtoHdfsProcessor) Destory ¶
func (p *LogtoHdfsProcessor) Destory() error
func (*LogtoHdfsProcessor) Process ¶
func (p *LogtoHdfsProcessor) Process(msg string) error
func (*LogtoHdfsProcessor) Tick ¶
func (p *LogtoHdfsProcessor) Tick() error
type MessageQueue ¶
type MessageQueue struct {
// contains filtered or unexported fields
}
message queue
func (*MessageQueue) AddQueue ¶
func (mq *MessageQueue) AddQueue(name, value string) error
add queue
type Processor ¶
type Processor interface { Init(ctx *sj.Json, id int, itemChans [](chan Item)) error Process(msg string) error Tick() error Destory() error }
func NewProcessor ¶
type TestCollector ¶
type TestCollector struct {
// contains filtered or unexported fields
}
func (*TestCollector) Collect ¶
func (p *TestCollector) Collect(item Item) error
func (*TestCollector) Destory ¶
func (p *TestCollector) Destory() error
func (*TestCollector) Tick ¶
func (p *TestCollector) Tick() error
type TestProcessor ¶
type TestProcessor struct {
// contains filtered or unexported fields
}
func (*TestProcessor) Destory ¶
func (p *TestProcessor) Destory() error
func (*TestProcessor) Process ¶
func (p *TestProcessor) Process(msg string) error
func (*TestProcessor) Tick ¶
func (p *TestProcessor) Tick() error
Click to show internal directories.
Click to hide internal directories.