goDB

package
v4.1.14 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Mar 11, 2025 License: GPL-2.0 Imports: 28 Imported by: 0

Documentation

Overview

Package goDB defines the columnar database format for persistently writing goProbe flows and querying the stored data.

Index

Constants

View Source 
const (

	// DBWriteInterval defines the periodic write out interval of goProbe
	DBWriteInterval int64 = 300

	// WorkBulkSize denotes the per-worker bulk size (number of GPDirs processed before
	// transmitting the resulting map to for further reduction / aggregtion
	WorkBulkSize = 32
)
View Source 
const DefaultPermissions = fs.FileMode(0644)

DefaultPermissions denotes the default permissions used during writeout

Variables

View Source 
var ErrIPVersionMismatch error

ErrIPVersionMismatch signifies that there is an IPv4 / IPv6 mismatch

Functions

This section is empty.

Types

type BulkWorkload 

type BulkWorkload struct {
	FlowMap      *hashmap.AggFlowMap
	CaptureStats capturetypes.CaptureStats
	Timestamp    int64
}

BulkWorkload denotes a set of workloads / writes to perform during WriteBulk()

type BytesRecStringParser 

type BytesRecStringParser struct{}

BytesRecStringParser parses bytes received counter strings

func (*BytesRecStringParser) ParseVal 

func (b *BytesRecStringParser) ParseVal(element string, val *types.Counters) error

ParseVal parses a number from a string and writes it to the "Byts Recevied" counter in val

type BytesSentStringParser 

type BytesSentStringParser struct{}

BytesSentStringParser parses bytes sent counter strings

func (*BytesSentStringParser) ParseVal 

func (b *BytesSentStringParser) ParseVal(element string, val *types.Counters) error

ParseVal parses a number from a string and writes it to the "Byts Sent" counter in val

type DBWorkManager 

type DBWorkManager struct {
	// contains filtered or unexported fields
}

DBWorkManager schedules parallel processing of blocks relevant for a query

func NewDBWorkManager 

func NewDBWorkManager(query *Query, dbpath string, iface string, numProcessingUnits int, opts ...WorkManagerOption) (*DBWorkManager, error)

NewDBWorkManager sets up a new work manager for executing queries

func (*DBWorkManager) Close 

func (w *DBWorkManager) Close()

Close releases all resources claimed by the DBWorkManager

func (*DBWorkManager) CreateWorkerJobs 

func (w *DBWorkManager) CreateWorkerJobs(tfirst int64, tlast int64) (nonempty bool, err error)

CreateWorkerJobs sets up all workloads for query execution

func (*DBWorkManager) ExecuteWorkerReadJobs 

func (w *DBWorkManager) ExecuteWorkerReadJobs(ctx context.Context, mapChan chan hashmap.AggFlowMapWithMetadata)

ExecuteWorkerReadJobs runs the query concurrently with multiple sprocessing units

func (*DBWorkManager) GetCoveredTimeInterval 

func (w *DBWorkManager) GetCoveredTimeInterval() (time.Time, time.Time)

GetCoveredTimeInterval can be used to determine the time span actually covered by the query

func (*DBWorkManager) GetNumWorkers 

func (w *DBWorkManager) GetNumWorkers() uint64

GetNumWorkers returns the number of workloads available to the outside world for loop bounds etc.

func (*DBWorkManager) ReadMetadata 

func (w *DBWorkManager) ReadMetadata(tfirst int64, tlast int64) (*InterfaceMetadata, error)

ReadMetadata extracts the metadata for a time range from the DB

type DBWriter 

type DBWriter struct {
	// contains filtered or unexported fields
}

DBWriter writes goProbe flows to goDB database files

func NewDBWriter 

func NewDBWriter(dbpath string, iface string, encoderType encoders.Type) (w *DBWriter)

NewDBWriter initializes a new DBWriter

func (*DBWriter) EncoderLevel 

func (w *DBWriter) EncoderLevel(level int) *DBWriter

EncoderLevel overrides the default encoder / compressor level for files / directories in the DB

func (*DBWriter) Permissions 

func (w *DBWriter) Permissions(permissions fs.FileMode) *DBWriter

Permissions overrides the default permissions for files / directories in the DB

func (*DBWriter) Write 

func (w *DBWriter) Write(flowmap *hashmap.AggFlowMap, captureStats capturetypes.CaptureStats, timestamp int64) error

Write takes an aggregated flow map and its metadata and writes it to disk for a given timestamp

func (*DBWriter) WriteBulk 

func (w *DBWriter) WriteBulk(workloads []BulkWorkload, dirTimestamp int64) (err error)

WriteBulk takes multiple aggregated flow maps and their metadata and writes it to disk for a given timestamp

type DIPStringParser 

type DIPStringParser struct{}

DIPStringParser parses dip strings

func (*DIPStringParser) ParseKey 

func (d *DIPStringParser) ParseKey(element string, key *types.ExtendedKey) error

ParseKey parses a destination IP string and writes it to the desintation IP key slice

type DportStringParser 

type DportStringParser struct{}

DportStringParser parses dport strings

func (*DportStringParser) ParseKey 

func (d *DportStringParser) ParseKey(element string, key *types.ExtendedKey) error

ParseKey parses a destination port string and writes it to the desintation port key slice

type FilterFn 

type FilterFn func(*hashmap.AggFlowMap) *hashmap.AggFlowMap

FilterFn denotes a function that filters an exisiting AggFlowMap and returns a new one Note: In case of a null-op, the output map may be the same as the input map

func QueryFilter 

func QueryFilter(query *Query) FilterFn

QueryFilter returns a FilterFn that applies a query condition to an existing AggFlowMap

type InterfaceMetadata 

type InterfaceMetadata struct {
	Iface string `json:"iface"`
	results.TimeRange

	gpfile.Stats
}

InterfaceMetadata describes the time range for which data is available, how many flows were recorded and how much traffic was captured

func (*InterfaceMetadata) TableHeader 

func (i *InterfaceMetadata) TableHeader(detailed bool) (headerRows [][]string)

TableHeader constructs the table header for pretty printing metadata

func (*InterfaceMetadata) TableRow 

func (i *InterfaceMetadata) TableRow(detailed bool) []string

TableRow puts all attributes of the metadata into a row that can be used for table printing. If detailed is false, the counts and metadata is summarized to their sum (e.g. IPv4 + IPv6 flows = NumFlows). Drops are only printed in detail mode

type NOPStringParser 

type NOPStringParser struct{}

NOPStringParser doesn't do anything and just lets everything through which is not understandable by the other attribute parsers (e.g. the % field or any other field not mentioned above)

func (*NOPStringParser) ParseKey 

func (n *NOPStringParser) ParseKey(_ string, _ *types.ExtendedKey) error

ParseKey is a no-op

func (*NOPStringParser) ParseVal 

func (n *NOPStringParser) ParseVal(_ string, _ *types.Counters) error

ParseVal is a no-op

type PacketsRecStringParser 

type PacketsRecStringParser struct{}

PacketsRecStringParser parses packets received counter strings

func (*PacketsRecStringParser) ParseVal 

func (p *PacketsRecStringParser) ParseVal(element string, val *types.Counters) error

ParseVal parses a number from a string and writes it to the "Packets Received" counter in val

type PacketsSentStringParser 

type PacketsSentStringParser struct{}

PacketsSentStringParser parses packets sent counter strings

func (*PacketsSentStringParser) ParseVal 

func (p *PacketsSentStringParser) ParseVal(element string, val *types.Counters) error

ParseVal parses a number from a string and writes it to the "Packets Sent" counter in val

type ProtoStringParser 

type ProtoStringParser struct{}

ProtoStringParser parses proto strings

func (*ProtoStringParser) ParseKey 

func (p *ProtoStringParser) ParseKey(element string, key *types.ExtendedKey) error

ParseKey parses an IP protocol string and writes it to the protocol key slice

type Query 

type Query struct {
	// list of attributes that will be compared, e.g. "dip" "sip"
	// in a "talk_conv" query
	Attributes  []types.Attribute
	Conditional node.Node

	sync.Mutex
	// contains filtered or unexported fields
}

Query stores all relevant parameters for data selection

func NewMetadataQuery 

func NewMetadataQuery() *Query

NewMetadataQuery creates a metadata-only query

func NewQuery 

func NewQuery(attributes []types.Attribute, conditional node.Node, selector types.LabelSelector) *Query

NewQuery creates a new Query object based on the parsed command line parameters

func (*Query) AttributesToString 

func (q *Query) AttributesToString() []string

AttributesToString is a convenience method for translating the query attributes into a human-readable name

func (*Query) IsLowMem 

func (q *Query) IsLowMem() bool

IsLowMem returns if the query was run in low-memory mode

func (*Query) Keepalive 

func (q *Query) Keepalive(fn func(), interval time.Duration) *Query

Keepalive enables sending keepalives at a given frequency

func (*Query) LowMem 

func (q *Query) LowMem(enable bool) *Query

LowMem enables memory-saving mode

func (*Query) UpdateKeepalive 

func (q *Query) UpdateKeepalive()

UpdateKeepalive emits a specific log line if enabled and a minimum time period has elapsed

type SIPStringParser 

type SIPStringParser struct{}

SIPStringParser parses sip strings

func (*SIPStringParser) ParseKey 

func (s *SIPStringParser) ParseKey(element string, key *types.ExtendedKey) error

ParseKey parses a source IP string and writes it to the source IP key slice

type StringKeyParser 

type StringKeyParser interface {
	ParseKey(element string, key *types.ExtendedKey) error
}

StringKeyParser is used for mapping a string to it's goDB key

func NewStringKeyParser 

func NewStringKeyParser(kind string) StringKeyParser

NewStringKeyParser selects a string parser based on the attribute

type StringValParser 

type StringValParser interface {
	ParseVal(element string, val *types.Counters) error
}

StringValParser is used for mapping a string to it's goDB value

func NewStringValParser 

func NewStringValParser(kind string) StringValParser

NewStringValParser selects a string parser based on a supported goDB counter

type SyslogDBWriter 

type SyslogDBWriter struct {
	// contains filtered or unexported fields
}

SyslogDBWriter can write goProbe's flow map to a syslog destination

func NewSyslogDBWriter 

func NewSyslogDBWriter() (*SyslogDBWriter, error)

NewSyslogDBWriter establishes a syslog connection and returns the flow writer

func (*SyslogDBWriter) Write 

func (s *SyslogDBWriter) Write(flowmap *hashmap.AggFlowMap, iface string, timestamp int64)

Write writes the aggregated flows to the syslog writer

type TimeStringParser 

type TimeStringParser struct{}

TimeStringParser parses time strings

func (*TimeStringParser) ParseKey 

func (t *TimeStringParser) ParseKey(element string, key *types.ExtendedKey) error

ParseKey parses a time string and writes it to the Time key

type WorkManagerOption 

type WorkManagerOption func(*DBWorkManager)

WorkManagerOption configures the DBWorkManager

Source Files

View all Source files

Directories

Path Synopsis
node
encoders
lz4
Package lz4 implements goDB's Encoder interface for lz4 (de-)compression of flow data
 Package lz4 implements goDB's Encoder interface for lz4 (de-)compression of flow data
lz4cust
Package lz4cust implements goDB's Encoder interface for lz4 (de-)compression of flow data
 Package lz4cust implements goDB's Encoder interface for lz4 (de-)compression of flow data
null
zstd
Package zstd implements goDB's Encoder interface for ZStandard (de-)compression of flow data
 Package zstd implements goDB's Encoder interface for ZStandard (de-)compression of flow data
benchgen
Package protocols provides lookup functionality for IP protocol IDs and their names (which are in some cases OS specific)
 Package protocols provides lookup functionality for IP protocol IDs and their names (which are in some cases OS specific)
gpfile

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.