duplicacy

package
v0.0.0-...-0a6b8aa Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 22, 2020 License: MIT Imports: 73 Imported by: 0

Documentation

Overview

Copyright (c) Acrosync LLC. All rights reserved. Free for personal use and commercial trial Commercial use requires per-user licenses available from https://duplicacy.com

Copyright (c) Acrosync LLC. All rights reserved. Free for personal use and commercial trial Commercial use requires per-user licenses available from https://duplicacy.com

This storage backend is based on the work by Yuri Karamani from https://github.com/karamani/webdavclnt, released under the MIT license.

Index

Constants

View Source 
const (
	ChunkOperationFind      = 0
	ChunkOperationDelete    = 1
	ChunkOperationFossilize = 2
	ChunkOperationResurrect = 3
)

These are operations that ChunkOperator will perform.

View Source 
const (
	DEBUG  = -2
	TRACE  = -1
	INFO   = 0
	WARN   = 1
	ERROR  = 2
	FATAL  = 3
	ASSERT = 4
)

Variables

View Source 
var (
	GCDFileMimeType      = "application/octet-stream"
	GCDDirectoryMimeType = "application/vnd.google-apps.folder"
	GCDUserDrive         = "root"
)
View Source 
var ACDRefreshTokenURL = "https://duplicacy.com/acd_refresh"
View Source 
var B2AuthorizationURL = "https://api.backblazeb2.com/b2api/v1/b2_authorize_account"
View Source 
var CONFIG_DEFAULT_ITERATIONS = 16384

The default iterations for key derivation

View Source 
var CONFIG_HEADER = "duplicacy\001"

The new header of the config file (to differentiate from the old format where the salt and iterations are fixed)

View Source 
var CONFIG_SALT_LENGTH = 32

The length of the salt used in the new format

View Source 
var DEFAULT_COMPRESSION_LEVEL = 100

The new default compression level is 100. However, in the early versions we use the standard zlib levels of -1 to 9.

View Source 
var DEFAULT_KEY = []byte("duplicacy")

If encryption is turned off, use this key for HMAC-SHA256 or chunk ID generation etc.

View Source 
var DUPLICACY_DIRECTORY = ".duplicacy"

This is the hidden directory in the repository for storing various files.

View Source 
var DUPLICACY_FILE = ".duplicacy"
View Source 
var DecryptWithHMACSHA256 = false
View Source 
var ENCRYPTION_HEADER = "duplicacy\000"

Magic word to identify a duplicacy format encrypted file, plus a version number.

View Source 
var ENCRYPTION_VERSION_RSA byte = 2

RSA encrypted chunks start with "duplicacy\002"

View Source 
var HubicCredentialURL = "https://api.hubic.com/1.0/account/credentials"
View Source 
var HubicRefreshTokenURL = "https://duplicacy.com/hubic_refresh"
View Source 
var LogFunction func(level int, logID string, message string)
View Source 
var MaskArray = [...]uint64{
	0, 0, 0, 0, 0, 0,
	0x00001803110,
	0x000018035100,
	0x00001800035300,
	0x000019000353000,
	0x0000590003530000,
	0x0000d90003530000,
	0x0000d90103530000,
	0x0000d90303530000,
	0x0000d90313530000,
	0x0000d90f03530000,
	0x0000d90303537000,
	0x0000d90703537000,
	0x0000d90707537000,
	0x0000d91707537000,
	0x0000d91747537000,
	0x0000d91767537000,
	0x0000d93767537000,
	0x0000d93777537000,
	0x0000d93777577000,
}
View Source 
var Preferences []Preference
View Source 
var RegexMap map[string]*regexp.Regexp
View Source 
var RunAtError func() = func() {}

This is the function to be called before exiting when an error occurs.

View Source 
var RunInBackground bool = false
View Source 
var SkipFileHash = false

Functions

func AllocateChunkBuffer 

func AllocateChunkBuffer() (buffer *bytes.Buffer)

func AppendPattern 

func AppendPattern(patterns []string, new_pattern string) (new_patterns []string)

func AtoSize 

func AtoSize(sizeString string) int

func B2Escape 

func B2Escape(path string) string

URL encode the given path but keep the slashes intact

func Benchmark 

func Benchmark(localDirectory string, storage Storage, fileSize int64, chunkSize int, chunkCount int, uploadThreads int, downloadThreads int) bool

func CatchLogException 

func CatchLogException()

func CloseSFTPStorage 

func CloseSFTPStorage(storage *SFTPStorage)

func ConfigStorage 

func ConfigStorage(storage Storage, iterations int, compressionLevel int, averageChunkSize int, maximumChunkSize int,
	minimumChunkSize int, password string, copyFrom *Config, bitCopy bool, keyFile string) bool

ConfigStorage makes the general storage space available for storing duplicacy format snapshots. In essence, it simply creates a file named 'config' that stores various parameters as well as a set of keys if encryption is enabled.

func CreateShadowCopy 

func CreateShadowCopy(top string, shadowCopy bool, timeoutInSeconds int) (shadowTop string)

func DeleteShadowCopy 

func DeleteShadowCopy()

func EnableLogHeader 

func EnableLogHeader()

func EnableStackTrace 

func EnableStackTrace()

func GenerateKeyFromPassword 

func GenerateKeyFromPassword(password string, salt []byte, iterations int) []byte

GenerateKeyFromPassword generates a key from the password.

func GetDuplicacyPreferencePath 

func GetDuplicacyPreferencePath() string

func GetOwner 

func GetOwner(entry *Entry, fileInfo *os.FileInfo)

func GetPassword 

func GetPassword(preference Preference, passwordType string, prompt string,
	showPassword bool, resetPassword bool) string

GetPassword attempts to get the password from KeyChain/KeyRing, environment variables, or keyboard input.

func GetPasswordFromPreference 

func GetPasswordFromPreference(preference Preference, passwordType string) string

Get password from preference, env, but don't start any keyring request

func IsDebugging 

func IsDebugging() bool

func IsEmptyFilter 

func IsEmptyFilter(pattern string) bool

func IsTracing 

func IsTracing() bool

func IsUnspecifiedFilter 

func IsUnspecifiedFilter(pattern string) bool

func IsValidRegex 

func IsValidRegex(pattern string) (valid bool, err error)

func LOG_ASSERT 

func LOG_ASSERT(logID string, format string, v ...interface{})

func LOG_DEBUG 

func LOG_DEBUG(logID string, format string, v ...interface{})

func LOG_ERROR 

func LOG_ERROR(logID string, format string, v ...interface{})

func LOG_FATAL 

func LOG_FATAL(logID string, format string, v ...interface{})

func LOG_INFO 

func LOG_INFO(logID string, format string, v ...interface{})

func LOG_TRACE 

func LOG_TRACE(logID string, format string, v ...interface{})

func LOG_WARN 

func LOG_WARN(logID string, format string, v ...interface{})

func LoadPreferences 

func LoadPreferences(repository string) bool

func MatchPath 

func MatchPath(filePath string, patterns []string) (included bool)

MatchPath returns 'true' if the file 'filePath' is excluded by the specified 'patterns'. Each pattern starts with either '+' or '-', whereas '-' indicates exclusion and '+' indicates inclusion. Wildcards like '*' and '?' may appear in the patterns. In case no matching pattern is found, the file will be excluded if all patterns are include patterns, and included otherwise.

func PrettyNumber 

func PrettyNumber(number int64) string

func PrettySize 

func PrettySize(size int64) string

func PrettyTime 

func PrettyTime(seconds int64) string

func ProcessFilterFile 

func ProcessFilterFile(patternFile string, includedFiles []string) (patterns []string)

func ProcessFilterLines 

func ProcessFilterLines(patternFileLines []string, includedFiles []string) (patterns []string)

func ProcessFilters 

func ProcessFilters(filtersFile string) (patterns []string)

func RateLimitedCopy 

func RateLimitedCopy(writer io.Writer, reader io.Reader, rate int) (written int64, err error)
func Readlink(path string) (isRegular bool, s string, err error)

func ReleaseChunkBuffer 

func ReleaseChunkBuffer(buffer *bytes.Buffer)

func RemoveIncompleteSnapshot 

func RemoveIncompleteSnapshot()

func SaveIncompleteSnapshot 

func SaveIncompleteSnapshot(snapshot *Snapshot)

SaveIncompleteSnapshot saves the incomplete snapshot under the preference directory

func SavePassword 

func SavePassword(preference Preference, passwordType string, password string)

SavePassword saves the specified password in the keyring/keychain.

func SavePreferences 

func SavePreferences() bool

func SetDuplicacyPreferencePath 

func SetDuplicacyPreferencePath(p string)

Normally 'preferencePath' is set in LoadPreferences; however, if LoadPreferences is not called, this function provide another change to set 'preferencePath'

func SetKeyringFile 

func SetKeyringFile(path string)

func SetLoggingLevel 

func SetLoggingLevel(level int)

func SetOwner 

func SetOwner(fullPath string, entry *Entry, fileInfo *os.FileInfo) bool

func SplitDir 

func SplitDir(fullPath string) (dir string, file string)

func SuppressLog 

func SuppressLog(id string)

func UploadConfig 

func UploadConfig(storage Storage, config *Config, password string, iterations int) bool

Types

type ACDClient 

type ACDClient struct {
	HTTPClient *http.Client

	TokenFile string
	Token     *oauth2.Token
	TokenLock *sync.Mutex

	ContentURL  string
	MetadataURL string

	TestMode bool
}

func NewACDClient 

func NewACDClient(tokenFile string) (*ACDClient, error)

func (*ACDClient) CreateDirectory 

func (client *ACDClient) CreateDirectory(parentID string, name string) (string, error)

func (*ACDClient) DeleteFile 

func (client *ACDClient) DeleteFile(fileID string) error

func (*ACDClient) DownloadFile 

func (client *ACDClient) DownloadFile(fileID string) (io.ReadCloser, int64, error)

func (*ACDClient) GetEndpoint 

func (client *ACDClient) GetEndpoint() (err error)

func (*ACDClient) ListByName 

func (client *ACDClient) ListByName(parentID string, name string) (string, bool, int64, error)

func (*ACDClient) ListEntries 

func (client *ACDClient) ListEntries(parentID string, listFiles bool, listDirectories bool) ([]ACDEntry, error)

func (*ACDClient) MoveFile 

func (client *ACDClient) MoveFile(fileID string, fromParentID string, toParentID string) error

func (*ACDClient) RefreshToken 

func (client *ACDClient) RefreshToken() (err error)

func (*ACDClient) UploadFile 

func (client *ACDClient) UploadFile(parentID string, name string, content []byte, rateLimit int) (fileID string, err error)

type ACDEntry 

type ACDEntry struct {
	Name string `json:"name"`
	ID   string `json:"id"`
	Size int64  `json:"size"`
	Kind string `json:"kind"`
}

type ACDError 

type ACDError struct {
	Status  int
	Message string `json:"message"`
}

func (ACDError) Error 

func (err ACDError) Error() string

type ACDGetEndpointOutput 

type ACDGetEndpointOutput struct {
	CustomerExists bool   `json:"customerExists"`
	ContentURL     string `json:"contentUrl"`
	MetadataURL    string `json:"metadataUrl"`
}

type ACDListEntriesOutput 

type ACDListEntriesOutput struct {
	Count     int        `json:"count"`
	NextToken string     `json:"nextToken"`
	Entries   []ACDEntry `json:"data"`
}

type ACDStorage 

type ACDStorage struct {
	StorageBase
	// contains filtered or unexported fields
}

func CreateACDStorage 

func CreateACDStorage(tokenFile string, storagePath string, threads int) (storage *ACDStorage, err error)

CreateACDStorage creates an ACD storage object.

func (*ACDStorage) CreateDirectory 

func (storage *ACDStorage) CreateDirectory(threadIndex int, dir string) (err error)

CreateDirectory creates a new directory.

func (*ACDStorage) DeleteFile 

func (storage *ACDStorage) DeleteFile(threadIndex int, filePath string) (err error)

DeleteFile deletes the file or directory at 'filePath'.

func (*ACDStorage) DownloadFile 

func (storage *ACDStorage) DownloadFile(threadIndex int, filePath string, chunk *Chunk) (err error)

DownloadFile reads the file at 'filePath' into the chunk.

func (*ACDStorage) EnableTestMode 

func (storage *ACDStorage) EnableTestMode()

Enable the test mode.

func (*ACDStorage) GetFileInfo 

func (storage *ACDStorage) GetFileInfo(threadIndex int, filePath string) (exist bool, isDir bool, size int64, err error)

GetFileInfo returns the information about the file or directory at 'filePath'.

func (*ACDStorage) IsCacheNeeded 

func (storage *ACDStorage) IsCacheNeeded() bool

If a local snapshot cache is needed for the storage to avoid downloading/uploading chunks too often when managing snapshots.

func (*ACDStorage) IsFastListing 

func (storage *ACDStorage) IsFastListing() bool

If the storage supports fast listing of files names.

func (*ACDStorage) IsMoveFileImplemented 

func (storage *ACDStorage) IsMoveFileImplemented() bool

If the 'MoveFile' method is implemented.

func (*ACDStorage) IsStrongConsistent 

func (storage *ACDStorage) IsStrongConsistent() bool

If the storage can guarantee strong consistency.

func (*ACDStorage) ListFiles 

func (storage *ACDStorage) ListFiles(threadIndex int, dir string) ([]string, []int64, error)

ListFiles return the list of files and subdirectories under 'dir' (non-recursively)

func (*ACDStorage) MoveFile 

func (storage *ACDStorage) MoveFile(threadIndex int, from string, to string) (err error)

MoveFile renames the file.

func (*ACDStorage) UploadFile 

func (storage *ACDStorage) UploadFile(threadIndex int, filePath string, content []byte) (err error)

UploadFile writes 'content' to the file at 'filePath'.

type AzureStorage 

type AzureStorage struct {
	StorageBase
	// contains filtered or unexported fields
}

func CreateAzureStorage 

func CreateAzureStorage(accountName string, accountKey string,
	containerName string, threads int) (azureStorage *AzureStorage, err error)

func (*AzureStorage) CreateDirectory 

func (storage *AzureStorage) CreateDirectory(threadIndex int, dir string) (err error)

CreateDirectory creates a new directory.

func (*AzureStorage) DeleteFile 

func (storage *AzureStorage) DeleteFile(threadIndex int, filePath string) (err error)

DeleteFile deletes the file or directory at 'filePath'.

func (*AzureStorage) DownloadFile 

func (storage *AzureStorage) DownloadFile(threadIndex int, filePath string, chunk *Chunk) (err error)

DownloadFile reads the file at 'filePath' into the chunk.

func (*AzureStorage) EnableTestMode 

func (storage *AzureStorage) EnableTestMode()

Enable the test mode.

func (*AzureStorage) GetFileInfo 

func (storage *AzureStorage) GetFileInfo(threadIndex int, filePath string) (exist bool, isDir bool, size int64, err error)

GetFileInfo returns the information about the file or directory at 'filePath'.

func (*AzureStorage) IsCacheNeeded 

func (storage *AzureStorage) IsCacheNeeded() bool

If a local snapshot cache is needed for the storage to avoid downloading/uploading chunks too often when managing snapshots.

func (*AzureStorage) IsFastListing 

func (storage *AzureStorage) IsFastListing() bool

If the storage supports fast listing of files names.

func (*AzureStorage) IsMoveFileImplemented 

func (storage *AzureStorage) IsMoveFileImplemented() bool

If the 'MoveFile' method is implemented.

func (*AzureStorage) IsStrongConsistent 

func (storage *AzureStorage) IsStrongConsistent() bool

If the storage can guarantee strong consistency.

func (*AzureStorage) ListFiles 

func (azureStorage *AzureStorage) ListFiles(threadIndex int, dir string) (files []string, sizes []int64, err error)

ListFiles return the list of files and subdirectories under 'dir' (non-recursively)

func (*AzureStorage) MoveFile 

func (storage *AzureStorage) MoveFile(threadIndex int, from string, to string) (err error)

MoveFile renames the file.

func (*AzureStorage) UploadFile 

func (storage *AzureStorage) UploadFile(threadIndex int, filePath string, content []byte) (err error)

UploadFile writes 'content' to the file at 'filePath'.

type B2AuthorizeAccountOutput 

type B2AuthorizeAccountOutput struct {
	AccountID          string
	AuthorizationToken string
	APIURL             string
	DownloadURL        string
}

type B2Client 

type B2Client struct {
	HTTPClient *http.Client

	AccountID        string
	ApplicationKeyID string
	ApplicationKey   string
	BucketName       string
	BucketID         string
	StorageDir       string

	Lock               sync.Mutex
	AuthorizationToken string
	APIURL             string
	DownloadURL        string
	IsAuthorized       bool

	UploadURLs   []string
	UploadTokens []string

	Threads        int
	MaximumRetries int
	TestMode       bool

	LastAuthorizationTime int64
}

func NewB2Client 

func NewB2Client(applicationKeyID string, applicationKey string, downloadURL string, storageDir string, threads int) *B2Client

func (*B2Client) AuthorizeAccount 

func (client *B2Client) AuthorizeAccount(threadIndex int) (err error, allowed bool)

func (*B2Client) DeleteFile 

func (client *B2Client) DeleteFile(threadIndex int, fileName string, fileID string) (err error)

func (*B2Client) DownloadFile 

func (client *B2Client) DownloadFile(threadIndex int, filePath string) (io.ReadCloser, int64, error)

func (*B2Client) FindBucket 

func (client *B2Client) FindBucket(bucketName string) (err error)

func (*B2Client) HideFile 

func (client *B2Client) HideFile(threadIndex int, fileName string) (fileID string, err error)

func (*B2Client) ListFileNames 

func (client *B2Client) ListFileNames(threadIndex int, startFileName string, singleFile bool, includeVersions bool) (files []*B2Entry, err error)

func (*B2Client) UploadFile 

func (client *B2Client) UploadFile(threadIndex int, filePath string, content []byte, rateLimit int) (err error)

type B2Entry 

type B2Entry struct {
	FileID          string
	FileName        string
	Action          string
	Size            int64
	UploadTimestamp int64
}

type B2Error 

type B2Error struct {
	Status  int
	Code    string
	Message string
}

func (*B2Error) Error 

func (err *B2Error) Error() string

type B2GetUploadArgumentOutput 

type B2GetUploadArgumentOutput struct {
	BucketID           string
	UploadURL          string
	AuthorizationToken string
}

type B2HideFileOutput 

type B2HideFileOutput struct {
	FileID string
}

type B2ListFileNamesOutput 

type B2ListFileNamesOutput struct {
	Files        []*B2Entry
	NextFileName string
	NextFileId   string
}

type B2Storage 

type B2Storage struct {
	StorageBase
	// contains filtered or unexported fields
}

func CreateB2Storage 

func CreateB2Storage(accountID string, applicationKey string, downloadURL string, bucket string, storageDir string, threads int) (storage *B2Storage, err error)

CreateB2Storage creates a B2 storage object.

func (*B2Storage) CreateDirectory 

func (storage *B2Storage) CreateDirectory(threadIndex int, dir string) (err error)

CreateDirectory creates a new directory.

func (*B2Storage) DeleteFile 

func (storage *B2Storage) DeleteFile(threadIndex int, filePath string) (err error)

DeleteFile deletes the file or directory at 'filePath'.

func (*B2Storage) DownloadFile 

func (storage *B2Storage) DownloadFile(threadIndex int, filePath string, chunk *Chunk) (err error)

DownloadFile reads the file at 'filePath' into the chunk.

func (*B2Storage) EnableTestMode 

func (storage *B2Storage) EnableTestMode()

Enable the test mode.

func (*B2Storage) GetFileInfo 

func (storage *B2Storage) GetFileInfo(threadIndex int, filePath string) (exist bool, isDir bool, size int64, err error)

GetFileInfo returns the information about the file or directory at 'filePath'.

func (*B2Storage) IsCacheNeeded 

func (storage *B2Storage) IsCacheNeeded() bool

If a local snapshot cache is needed for the storage to avoid downloading/uploading chunks too often when managing snapshots.

func (*B2Storage) IsFastListing 

func (storage *B2Storage) IsFastListing() bool

If the storage supports fast listing of files names.

func (*B2Storage) IsMoveFileImplemented 

func (storage *B2Storage) IsMoveFileImplemented() bool

If the 'MoveFile' method is implemented.

func (*B2Storage) IsStrongConsistent 

func (storage *B2Storage) IsStrongConsistent() bool

If the storage can guarantee strong consistency.

func (*B2Storage) ListFiles 

func (storage *B2Storage) ListFiles(threadIndex int, dir string) (files []string, sizes []int64, err error)

ListFiles return the list of files and subdirectories under 'dir' (non-recursively)

func (*B2Storage) MoveFile 

func (storage *B2Storage) MoveFile(threadIndex int, from string, to string) (err error)

MoveFile renames the file.

func (*B2Storage) UploadFile 

func (storage *B2Storage) UploadFile(threadIndex int, filePath string, content []byte) (err error)

UploadFile writes 'content' to the file at 'filePath'.

type B2UploadArgument 

type B2UploadArgument struct {
	URL   string
	Token string
}

type BackupManager 

type BackupManager struct {
	SnapshotManager *SnapshotManager // the snapshot manager
	// contains filtered or unexported fields
}

func CreateBackupManager 

func CreateBackupManager(snapshotID string, storage Storage, top string, password string, nobackupFile string, filtersFile string) *BackupManager

CreateBackupManager creates a backup manager using the specified 'storage'. 'snapshotID' is a unique id to identify snapshots created for this repository. 'top' is the top directory of the repository. 'password' is the master key which can be nil if encryption is not enabled.

func (*BackupManager) Backup 

func (manager *BackupManager) Backup(top string, quickMode bool, threads int, tag string,
	showStatistics bool, shadowCopy bool, shadowCopyTimeout int, enumOnly bool) bool

Backup creates a snapshot for the repository 'top'. If 'quickMode' is true, only files with different sizes or timestamps since last backup will be uploaded (however the snapshot is still a full snapshot that shares unmodified files with last backup). Otherwise (or if this is the first backup), the entire repository will be scanned to create the snapshot. 'tag' is the tag assigned to the new snapshot.

func (*BackupManager) CopySnapshots 

func (manager *BackupManager) CopySnapshots(otherManager *BackupManager, snapshotID string,
	revisionsToBeCopied []int, threads int) bool

CopySnapshots copies the specified snapshots from one storage to the other.

func (*BackupManager) LoadRSAPrivateKey 

func (manager *BackupManager) LoadRSAPrivateKey(keyFile string, passphrase string)

loadRSAPrivateKey loads the specifed private key file for decrypting file chunks

func (*BackupManager) Restore 

func (manager *BackupManager) Restore(top string, revision int, inPlace bool, quickMode bool, threads int, overwrite bool,
	deleteMode bool, setOwner bool, showStatistics bool, patterns []string) bool

Restore downloads the specified snapshot, compares it with what's on the repository, and then downloads files that are different. 'base' is a directory that contains files at a different revision which can serve as a local cache to avoid download chunks available locally. It is perfectly ok for 'base' to be the same as 'top'. 'quickMode' will bypass files with unchanged sizes and timestamps. 'deleteMode' will remove local files that don't exist in the snapshot. 'patterns' is used to include/exclude certain files.

func (*BackupManager) RestoreFile 

func (manager *BackupManager) RestoreFile(chunkDownloader *ChunkDownloader, chunkMaker *ChunkMaker, entry *Entry, top string, inPlace bool, overwrite bool,
	showStatistics bool, totalFileSize int64, downloadedFileSize int64, startTime int64) bool

Restore downloads a file from the storage. If 'inPlace' is false, the download file is saved first to a temporary file under the .duplicacy directory and then replaces the existing one. Otherwise, the existing file will be overwritten directly.

func (*BackupManager) SetDryRun 

func (manager *BackupManager) SetDryRun(dryRun bool)

func (*BackupManager) SetupSnapshotCache 

func (manager *BackupManager) SetupSnapshotCache(storageName string) bool

SetupSnapshotCache creates the snapshot cache, which is merely a local storage under the default .duplicacy directory

func (*BackupManager) UploadSnapshot 

func (manager *BackupManager) UploadSnapshot(chunkMaker *ChunkMaker, uploader *ChunkUploader, top string, snapshot *Snapshot,
	chunkCache map[string]bool) (totalSnapshotChunkSize int64,
	numberOfNewSnapshotChunks int, totalUploadedSnapshotChunkSize int64,
	totalUploadedSnapshotChunkBytes int64)

UploadSnapshot uploads the specified snapshot to the storage. It turns Files, ChunkHashes, and ChunkLengths into sequences of chunks, and uploads these chunks, and finally the snapshot file.

type ByChunk 

type ByChunk []*Entry

This is used to sort entries by their starting chunks (and starting offsets if the starting chunks are the same).

func (ByChunk) Len 

func (entries ByChunk) Len() int

func (ByChunk) Less 

func (entries ByChunk) Less(i, j int) bool

func (ByChunk) Swap 

func (entries ByChunk) Swap(i, j int)

type ByName 

type ByName []*Entry

This is used to sort entries by their names.

func (ByName) Len 

func (entries ByName) Len() int

func (ByName) Less 

func (entries ByName) Less(i, j int) bool

func (ByName) Swap 

func (entries ByName) Swap(i, j int)

type Chunk 

type Chunk struct {
	// contains filtered or unexported fields
}

Chunk is the object being passed between the chunk maker, the chunk uploader, and chunk downloader. It can be read and written like a bytes.Buffer, and provides convenient functions to calculate the hash and id of the chunk.

func CreateChunk 

func CreateChunk(config *Config, bufferNeeded bool) *Chunk

CreateChunk creates a new chunk.

func (*Chunk) Decrypt 

func (chunk *Chunk) Decrypt(encryptionKey []byte, derivationKey string) (err error)

Decrypt decrypts the encrypted data stored in the chunk buffer. If derivationKey is not nil, the actual encryption key will be HMAC-SHA256(encryptionKey, derivationKey).

func (*Chunk) Encrypt 

func (chunk *Chunk) Encrypt(encryptionKey []byte, derivationKey string, isSnapshot bool) (err error)

Encrypt encrypts the plain data stored in the chunk buffer. If derivationKey is not nil, the actual encryption key will be HMAC-SHA256(encryptionKey, derivationKey).

func (*Chunk) GetBytes 

func (chunk *Chunk) GetBytes() []byte

GetBytes returns data available in this chunk

func (*Chunk) GetHash 

func (chunk *Chunk) GetHash() string

GetHash returns the chunk hash.

func (*Chunk) GetID 

func (chunk *Chunk) GetID() string

GetID returns the chunk id.

func (*Chunk) GetLength 

func (chunk *Chunk) GetLength() int

GetLength returns the length of available data

func (*Chunk) Reset 

func (chunk *Chunk) Reset(hashNeeded bool)

Reset makes the chunk reusable by clearing the existing data in the buffers. 'hashNeeded' indicates whether the hash of the new data to be read is needed. If the data to be read in is encrypted, there is no need to calculate the hash so hashNeeded should be 'false'.

func (*Chunk) VerifyID 

func (chunk *Chunk) VerifyID()

func (*Chunk) Write 

func (chunk *Chunk) Write(p []byte) (int, error)

Write implements the Writer interface.

type ChunkDownloadCompletion 

type ChunkDownloadCompletion struct {
	// contains filtered or unexported fields
}

ChunkDownloadCompletion represents the nofication when a chunk has been downloaded.

type ChunkDownloadTask 

type ChunkDownloadTask struct {
	// contains filtered or unexported fields
}

ChunkDownloadTask encapsulates information need to download a chunk.

type ChunkDownloader 

type ChunkDownloader struct {
	// contains filtered or unexported fields
}

ChunkDownloader is capable of performing multi-threaded downloading. Chunks to be downloaded are first organized as a list of ChunkDownloadTasks, with only the chunkHash field initialized. When a chunk is needed, the corresponding ChunkDownloadTask is sent to the dowloading goroutine. Once a chunk is downloaded, it will be inserted in the completed task list.

func CreateChunkDownloader 

func CreateChunkDownloader(config *Config, storage Storage, snapshotCache *FileStorage, showStatistics bool, threads int) *ChunkDownloader

func (*ChunkDownloader) AddChunk 

func (downloader *ChunkDownloader) AddChunk(chunkHash string) int

AddChunk adds a single chunk the download list.

func (*ChunkDownloader) AddFiles 

func (downloader *ChunkDownloader) AddFiles(snapshot *Snapshot, files []*Entry)

AddFiles adds chunks needed by the specified files to the download list.

func (*ChunkDownloader) Download 

func (downloader *ChunkDownloader) Download(threadIndex int, task ChunkDownloadTask) bool

Download downloads a chunk from the storage.

func (*ChunkDownloader) GetLastDownloadedChunk 

func (downloader *ChunkDownloader) GetLastDownloadedChunk() (chunk *Chunk, chunkHash string)

Return the chunk last downloaded and its hash

func (*ChunkDownloader) Prefetch 

func (downloader *ChunkDownloader) Prefetch(file *Entry)

Prefetch adds up to 'threads' chunks needed by a file to the download list

func (*ChunkDownloader) Reclaim 

func (downloader *ChunkDownloader) Reclaim(chunkIndex int)

Reclaim releases the downloaded chunk to the chunk pool

func (*ChunkDownloader) Stop 

func (downloader *ChunkDownloader) Stop()

Stop terminates all downloading goroutines

func (*ChunkDownloader) WaitForChunk 

func (downloader *ChunkDownloader) WaitForChunk(chunkIndex int) (chunk *Chunk)

WaitForChunk waits until the specified chunk is ready

func (*ChunkDownloader) WaitForCompletion 

func (downloader *ChunkDownloader) WaitForCompletion()

WaitForCompletion waits until all chunks have been downloaded

type ChunkMaker 

type ChunkMaker struct {
	// contains filtered or unexported fields
}

ChunkMaker breaks data into chunks using fastcdc. To save memory, the chunk maker only use a circular buffer whose size is double the minimum chunk size.

func CreateChunkMaker 

func CreateChunkMaker(config *Config, hashOnly bool) *ChunkMaker

CreateChunkMaker creates a chunk maker. 'randomSeed' is used to generate the character-to-integer table needed by fastcdc.

func (*ChunkMaker) ForEachChunk 

func (maker *ChunkMaker) ForEachChunk(reader io.Reader, endOfChunk func(chunk *Chunk, final bool),
	nextReader func(size int64, hash string) (io.Reader, bool))

ForEachChunk reads data from 'reader'. If EOF is encountered, it will call 'nextReader' to ask for next file. If 'nextReader' returns false, it will process remaining data in the buffer and then quit. When a chunk is identified, it will call 'endOfChunk' to return the chunk size and a boolean flag indicating if it is the last chunk.

type ChunkOperator 

type ChunkOperator struct {
	// contains filtered or unexported fields
}

ChunkOperator is capable of performing multi-threaded operations on chunks.

func CreateChunkOperator 

func CreateChunkOperator(storage Storage, threads int) *ChunkOperator

CreateChunkOperator creates a new ChunkOperator.

func (*ChunkOperator) AddTask 

func (operator *ChunkOperator) AddTask(operation int, chunkID string, filePath string)

func (*ChunkOperator) Delete 

func (operator *ChunkOperator) Delete(chunkID string, filePath string)

func (*ChunkOperator) Find 

func (operator *ChunkOperator) Find(chunkID string)

func (*ChunkOperator) Fossilize 

func (operator *ChunkOperator) Fossilize(chunkID string, filePath string)

func (*ChunkOperator) Resurrect 

func (operator *ChunkOperator) Resurrect(chunkID string, filePath string)

func (*ChunkOperator) Run 

func (operator *ChunkOperator) Run(threadIndex int, task ChunkOperatorTask)

func (*ChunkOperator) Stop 

func (operator *ChunkOperator) Stop()

type ChunkOperatorTask 

type ChunkOperatorTask struct {
	// contains filtered or unexported fields
}

ChunkOperatorTask is used to pass parameters for different kinds of chunk operations.

type ChunkUploadTask 

type ChunkUploadTask struct {
	// contains filtered or unexported fields
}

ChunkUploadTask represents a chunk to be uploaded.

type ChunkUploader 

type ChunkUploader struct {
	// contains filtered or unexported fields
}

ChunkUploader uploads chunks to the storage using one or more uploading goroutines. Chunks are added by the call to StartChunk(), and then passed to the uploading goroutines. The completion function is called when the downloading is completed. Note that ChunkUploader does not release chunks to the chunk pool; instead

func CreateChunkUploader 

func CreateChunkUploader(config *Config, storage Storage, snapshotCache *FileStorage, threads int,
	completionFunc func(chunk *Chunk, chunkIndex int, skipped bool, chunkSize int, uploadSize int)) *ChunkUploader

CreateChunkUploader creates a chunk uploader.

func (*ChunkUploader) Start 

func (uploader *ChunkUploader) Start()

Starts starts uploading goroutines.

func (*ChunkUploader) StartChunk 

func (uploader *ChunkUploader) StartChunk(chunk *Chunk, chunkIndex int)

StartChunk sends a chunk to be uploaded to a waiting uploading goroutine. It may block if all uploading goroutines are busy.

func (*ChunkUploader) Stop 

func (uploader *ChunkUploader) Stop()

Stop stops all uploading goroutines.

func (*ChunkUploader) Upload 

func (uploader *ChunkUploader) Upload(threadIndex int, task ChunkUploadTask) bool

Upload is called by the uploading goroutines to perform the actual uploading

type Config 

type Config struct {
	CompressionLevel int `json:"compression-level"`
	AverageChunkSize int `json:"average-chunk-size"`
	MaximumChunkSize int `json:"max-chunk-size"`
	MinimumChunkSize int `json:"min-chunk-size"`

	ChunkSeed []byte `json:"chunk-seed"`

	FixedNesting bool `json:"fixed-nesting"`

	// the HMAC-SHA256 key of the chunk data
	HashKey []byte `json:"-"`

	// used to generate an id from the chunk hash
	IDKey []byte `json:"-"`

	// for encrypting a chunk
	ChunkKey []byte `json:"-"`

	// for encrypting a non-chunk file
	FileKey []byte `json:"-"`
	// contains filtered or unexported fields
}

func CreateConfig 

func CreateConfig() (config *Config)

func CreateConfigFromParameters 

func CreateConfigFromParameters(compressionLevel int, averageChunkSize int, maximumChunkSize int, mininumChunkSize int,
	isEncrypted bool, copyFrom *Config, bitCopy bool) (config *Config)

func DownloadConfig 

func DownloadConfig(storage Storage, password string) (config *Config, isEncrypted bool, err error)

func (*Config) ComputeFileHash 

func (config *Config) ComputeFileHash(path string, buffer []byte) string

Calculate the file hash using the corresponding hasher

func (*Config) GetChunk 

func (config *Config) GetChunk() (chunk *Chunk)

func (*Config) GetChunkIDFromHash 

func (config *Config) GetChunkIDFromHash(hash string) string

GetChunkIDFromHash creates a chunk id from the chunk hash. The chunk id will be used as the name of the chunk file, so it is publicly exposed. The chunk hash is the HMAC-SHA256 of what is contained in the chunk and should never be exposed.

func (*Config) IsCompatiableWith 

func (config *Config) IsCompatiableWith(otherConfig *Config) bool

func (*Config) MarshalJSON 

func (config *Config) MarshalJSON() ([]byte, error)

func (*Config) NewFileHasher 

func (config *Config) NewFileHasher() hash.Hash

func (*Config) NewKeyedHasher 

func (config *Config) NewKeyedHasher(key []byte) hash.Hash

func (*Config) Print 

func (config *Config) Print()

func (*Config) PutChunk 

func (config *Config) PutChunk(chunk *Chunk)

func (*Config) UnmarshalJSON 

func (config *Config) UnmarshalJSON(description []byte) (err error)

type DropboxStorage 

type DropboxStorage struct {
	StorageBase
	// contains filtered or unexported fields
}

func CreateDropboxStorage 

func CreateDropboxStorage(accessToken string, storageDir string, minimumNesting int, threads int) (storage *DropboxStorage, err error)

CreateDropboxStorage creates a dropbox storage object.

func (*DropboxStorage) CreateDirectory 

func (storage *DropboxStorage) CreateDirectory(threadIndex int, dir string) (err error)

CreateDirectory creates a new directory.

func (*DropboxStorage) DeleteFile 

func (storage *DropboxStorage) DeleteFile(threadIndex int, filePath string) (err error)

DeleteFile deletes the file or directory at 'filePath'.

func (*DropboxStorage) DownloadFile 

func (storage *DropboxStorage) DownloadFile(threadIndex int, filePath string, chunk *Chunk) (err error)

DownloadFile reads the file at 'filePath' into the chunk.

func (*DropboxStorage) EnableTestMode 

func (storage *DropboxStorage) EnableTestMode()

Enable the test mode.

func (*DropboxStorage) GetFileInfo 

func (storage *DropboxStorage) GetFileInfo(threadIndex int, filePath string) (exist bool, isDir bool, size int64, err error)

GetFileInfo returns the information about the file or directory at 'filePath'.

func (*DropboxStorage) IsCacheNeeded 

func (storage *DropboxStorage) IsCacheNeeded() bool

If a local snapshot cache is needed for the storage to avoid downloading/uploading chunks too often when managing snapshots.

func (*DropboxStorage) IsFastListing 

func (storage *DropboxStorage) IsFastListing() bool

If the storage supports fast listing of files names.

func (*DropboxStorage) IsMoveFileImplemented 

func (storage *DropboxStorage) IsMoveFileImplemented() bool

If the 'MoveFile' method is implemented.

func (*DropboxStorage) IsStrongConsistent 

func (storage *DropboxStorage) IsStrongConsistent() bool

If the storage can guarantee strong consistency.

func (*DropboxStorage) ListFiles 

func (storage *DropboxStorage) ListFiles(threadIndex int, dir string) (files []string, sizes []int64, err error)

ListFiles return the list of files and subdirectories under 'dir' (non-recursively)

func (*DropboxStorage) MoveFile 

func (storage *DropboxStorage) MoveFile(threadIndex int, from string, to string) (err error)

MoveFile renames the file.

func (*DropboxStorage) UploadFile 

func (storage *DropboxStorage) UploadFile(threadIndex int, filePath string, content []byte) (err error)

UploadFile writes 'content' to the file at 'filePath'.

type DummyHasher 

type DummyHasher struct {
}

Implement a dummy hasher to be used when SkipFileHash is true.

func (*DummyHasher) BlockSize 

func (hasher *DummyHasher) BlockSize() int

func (*DummyHasher) Reset 

func (hasher *DummyHasher) Reset()

func (*DummyHasher) Size 

func (hasher *DummyHasher) Size() int

func (*DummyHasher) Sum 

func (hasher *DummyHasher) Sum(b []byte) []byte

func (*DummyHasher) Write 

func (hasher *DummyHasher) Write(p []byte) (int, error)

type Entry 

type Entry struct {
	Path string
	Size int64
	Time int64
	Mode uint32
	Link string
	Hash string

	UID int
	GID int

	StartChunk  int
	StartOffset int
	EndChunk    int
	EndOffset   int

	Attributes map[string][]byte
}

Entry encapsulates information about a file or directory.

func CreateEntry 

func CreateEntry(path string, size int64, time int64, mode uint32) *Entry

CreateEntry creates an entry from file properties.

func CreateEntryFromFileInfo 

func CreateEntryFromFileInfo(fileInfo os.FileInfo, directory string) *Entry

CreateEntryFromFileInfo creates an entry from a 'FileInfo' object.

func ListEntries 

func ListEntries(top string, path string, fileList *[]*Entry, patterns []string, nobackupFile string, discardAttributes bool) (directoryList []*Entry,
	skippedFiles []string, err error)

ListEntries returns a list of entries representing file and subdirectories under the directory 'path'. Entry paths are normalized as relative to 'top'. 'patterns' are used to exclude or include certain files.

func (*Entry) Compare 

func (left *Entry) Compare(right *Entry) int

Return -1 if 'left' should appear before 'right', 1 if opposite, and 0 if they are the same. Files are always arranged before subdirectories under the same parent directory.

func (*Entry) Diff 

func (entry *Entry) Diff(chunkHashes []string, chunkLengths []int,
	otherHashes []string, otherLengths []int) (modifiedLength int64)

Diff returns how many bytes remain unmodifiled between two files.

func (*Entry) GetPermissions 

func (entry *Entry) GetPermissions() os.FileMode

func (*Entry) IsDir 

func (entry *Entry) IsDir() bool

func (*Entry) IsFile 

func (entry *Entry) IsFile() bool
func (entry *Entry) IsLink() bool

func (*Entry) IsSameAs 

func (entry *Entry) IsSameAs(other *Entry) bool

func (*Entry) IsSameAsFileInfo 

func (entry *Entry) IsSameAsFileInfo(other os.FileInfo) bool

func (*Entry) MarshalJSON 

func (entry *Entry) MarshalJSON() ([]byte, error)

MarshalJSON returns the json description of an entry.

func (*Entry) ReadAttributes 

func (entry *Entry) ReadAttributes(top string)

func (*Entry) RestoreMetadata 

func (entry *Entry) RestoreMetadata(fullPath string, fileInfo *os.FileInfo, setOwner bool) bool

func (*Entry) SetAttributesToFile 

func (entry *Entry) SetAttributesToFile(fullPath string)

func (*Entry) String 

func (entry *Entry) String(maxSizeDigits int) string

func (*Entry) UnmarshalJSON 

func (entry *Entry) UnmarshalJSON(description []byte) (err error)

CreateEntryFromJSON creates an entry from a json description.

type Exception 

type Exception struct {
	Level   int
	LogID   string
	Message string
}

type FileFabricFile 

type FileFabricFile struct {
	XMLName xml.Name
	ID      string `xml:"fi_id"`
	Path    string `xml:"path"`
	Size    int64  `xml:"fi_size"`
	Type    int    `xml:"fi_type"`
}

The XML element representing a file returned by the File Fabric server

type FileFabricFileList 

type FileFabricFileList struct {
	XMLName xml.Name         `xml:"files"`
	Files   []FileFabricFile `xml:",any"`
}

The XML element representing a file list returned by the server

type FileFabricResponse 

type FileFabricResponse struct {
	Status  string `xml:"status"`
	Message string `xml:"statusmessage"`
}

The general server response

type FileFabricStorage 

type FileFabricStorage struct {
	StorageBase
	// contains filtered or unexported fields
}

func CreateFileFabricStorage 

func CreateFileFabricStorage(endpoint string, token string, storageDir string, threads int) (storage *FileFabricStorage, err error)

Create a File Fabric storage backend

func (*FileFabricStorage) CreateDirectory 

func (storage *FileFabricStorage) CreateDirectory(threadIndex int, dir string) (err error)

func (*FileFabricStorage) DeleteFile 

func (storage *FileFabricStorage) DeleteFile(threadIndex int, filePath string) (err error)

DeleteFile deletes the file or directory at 'filePath'.

func (*FileFabricStorage) DownloadFile 

func (storage *FileFabricStorage) DownloadFile(threadIndex int, filePath string, chunk *Chunk) (err error)

DownloadFile reads the file at 'filePath' into the chunk.

func (*FileFabricStorage) EnableTestMode 

func (storage *FileFabricStorage) EnableTestMode()

Enable the test mode.

func (*FileFabricStorage) GetFileInfo 

func (storage *FileFabricStorage) GetFileInfo(threadIndex int, filePath string) (exist bool, isDir bool, size int64, err error)

GetFileInfo returns the information about the file or directory at 'filePath'. This is a function required by the Storage interface.

func (*FileFabricStorage) IsCacheNeeded 

func (storage *FileFabricStorage) IsCacheNeeded() bool

If a local snapshot cache is needed for the storage to avoid downloading/uploading chunks too often when managing snapshots.

func (*FileFabricStorage) IsFastListing 

func (storage *FileFabricStorage) IsFastListing() bool

If the storage supports fast listing of files names.

func (*FileFabricStorage) IsMoveFileImplemented 

func (storage *FileFabricStorage) IsMoveFileImplemented() bool

If the 'MoveFile' method is implemented.

func (*FileFabricStorage) IsStrongConsistent 

func (storage *FileFabricStorage) IsStrongConsistent() bool

If the storage can guarantee strong consistency.

func (*FileFabricStorage) ListFiles 

func (storage *FileFabricStorage) ListFiles(threadIndex int, dir string) (files []string, sizes []int64, err error)

ListFiles return the list of files and subdirectories under 'dir'. A subdirectories returned must have a trailing '/', with a size of 0. If 'dir' is 'snapshots', only subdirectories will be returned. If 'dir' is 'snapshots/repository_id', then only files will be returned. If 'dir' is 'chunks', the implementation can return the list either recusively or non-recusively.

func (*FileFabricStorage) MoveFile 

func (storage *FileFabricStorage) MoveFile(threadIndex int, from string, to string) (err error)

MoveFile renames the file.

func (*FileFabricStorage) UploadFile 

func (storage *FileFabricStorage) UploadFile(threadIndex int, filePath string, content []byte) (err error)

UploadFile writes 'content' to the file at 'filePath'.

type FileInfoCompare 

type FileInfoCompare []os.FileInfo

This is used to sort FileInfo objects.

func (FileInfoCompare) Len 

func (files FileInfoCompare) Len() int

func (FileInfoCompare) Less 

func (files FileInfoCompare) Less(i, j int) bool

func (FileInfoCompare) Swap 

func (files FileInfoCompare) Swap(i, j int)

type FileReader 

type FileReader struct {
	CurrentFile  *os.File
	CurrentIndex int
	CurrentEntry *Entry

	SkippedFiles []string
	// contains filtered or unexported fields
}

FileReader wraps a number of files and turns them into a series of readers.

func CreateFileReader 

func CreateFileReader(top string, files []*Entry) *FileReader

CreateFileReader creates a file reader.

func (*FileReader) NextFile 

func (reader *FileReader) NextFile() bool

NextFile switches to the next file in the file reader.

type FileStorage 

type FileStorage struct {
	StorageBase
	// contains filtered or unexported fields
}

FileStorage is a local on-disk file storage implementing the Storage interface.

func CreateFileStorage 

func CreateFileStorage(storageDir string, isCacheNeeded bool, threads int) (storage *FileStorage, err error)

CreateFileStorage creates a file storage.

func (*FileStorage) CreateDirectory 

func (storage *FileStorage) CreateDirectory(threadIndex int, dir string) (err error)

CreateDirectory creates a new directory.

func (*FileStorage) DeleteFile 

func (storage *FileStorage) DeleteFile(threadIndex int, filePath string) (err error)

DeleteFile deletes the file or directory at 'filePath'.

func (*FileStorage) DownloadFile 

func (storage *FileStorage) DownloadFile(threadIndex int, filePath string, chunk *Chunk) (err error)

DownloadFile reads the file at 'filePath' into the chunk.

func (*FileStorage) EnableTestMode 

func (storage *FileStorage) EnableTestMode()

Enable the test mode.

func (*FileStorage) GetFileInfo 

func (storage *FileStorage) GetFileInfo(threadIndex int, filePath string) (exist bool, isDir bool, size int64, err error)

GetFileInfo returns the information about the file or directory at 'filePath'.

func (*FileStorage) IsCacheNeeded 

func (storage *FileStorage) IsCacheNeeded() bool

If a local snapshot cache is needed for the storage to avoid downloading/uploading chunks too often when managing snapshots.

func (*FileStorage) IsFastListing 

func (storage *FileStorage) IsFastListing() bool

If the storage supports fast listing of files names.

func (*FileStorage) IsMoveFileImplemented 

func (storage *FileStorage) IsMoveFileImplemented() bool

If the 'MoveFile' method is implemented.

func (*FileStorage) IsStrongConsistent 

func (storage *FileStorage) IsStrongConsistent() bool

If the storage can guarantee strong consistency.

func (*FileStorage) ListFiles 

func (storage *FileStorage) ListFiles(threadIndex int, dir string) (files []string, sizes []int64, err error)

ListFiles return the list of files and subdirectories under 'dir' (non-recursively).

func (*FileStorage) MoveFile 

func (storage *FileStorage) MoveFile(threadIndex int, from string, to string) (err error)

MoveFile renames the file.

func (*FileStorage) UploadFile 

func (storage *FileStorage) UploadFile(threadIndex int, filePath string, content []byte) (err error)

UploadFile writes 'content' to the file at 'filePath'

type FossilCollection 

type FossilCollection struct {

	// At what time the fossil collection was finished
	EndTime int64 `json:"end_time"`

	// The lastest revision for each snapshot id when the fossil collection was created.
	LastRevisions map[string]int `json:"last_revisions"`

	// Record the set of snapshots that have been removed by the prune command that created this fossil collection
	DeletedRevisions map[string][]int `json:"deleted_revisions"`

	// Fossils (i.e., chunks not referenced by any snapshots)
	Fossils []string `json:"fossils"`

	// Temporary files.
	Temporaries []string `json:"temporaries"`
}

FossilCollection contains fossils and temporary files found during a snapshot deletions.

func CreateFossilCollection 

func CreateFossilCollection(allSnapshots map[string][]*Snapshot) *FossilCollection

CreateFossilCollection creates an empty fossil collection

func (*FossilCollection) AddFossil 

func (collection *FossilCollection) AddFossil(hash string)

func (*FossilCollection) AddTemporary 

func (collection *FossilCollection) AddTemporary(temporary string)

func (*FossilCollection) IsDeletable 

func (collection *FossilCollection) IsDeletable(isStrongConsistent bool, ignoredIDs []string,
	allSnapshots map[string][]*Snapshot) (isDeletable bool, newSnapshots []*Snapshot)

IsDeletable determines if the previously collected fossils are safe to be permanently removed. If so, it will also returns a number of snapshots that were created during or after these fossils were being collected. Therefore, some fossils may be referenced by these new snapshots and they must be resurrected.

func (*FossilCollection) IsEmpty 

func (collection *FossilCollection) IsEmpty() bool

type GCDConfig 

type GCDConfig struct {
	ClientID     string          `json:"client_id"`
	ClientSecret string          `json:"client_secret"`
	Endpoint     oauth2.Endpoint `json:"end_point"`
	Token        oauth2.Token    `json:"token"`
}

type GCDStorage 

type GCDStorage struct {
	StorageBase

	TestMode bool
	// contains filtered or unexported fields
}

func CreateGCDStorage 

func CreateGCDStorage(tokenFile string, driveID string, storagePath string, threads int) (storage *GCDStorage, err error)

CreateGCDStorage creates a GCD storage object.

func (*GCDStorage) CreateDirectory 

func (storage *GCDStorage) CreateDirectory(threadIndex int, dir string) (err error)

createDirectory creates a new directory.

func (*GCDStorage) DeleteFile 

func (storage *GCDStorage) DeleteFile(threadIndex int, filePath string) (err error)

DeleteFile deletes the file or directory at 'filePath'.

func (*GCDStorage) DownloadFile 

func (storage *GCDStorage) DownloadFile(threadIndex int, filePath string, chunk *Chunk) (err error)

DownloadFile reads the file at 'filePath' into the chunk.

func (*GCDStorage) EnableTestMode 

func (storage *GCDStorage) EnableTestMode()

Enable the test mode.

func (*GCDStorage) GetFileInfo 

func (storage *GCDStorage) GetFileInfo(threadIndex int, filePath string) (exist bool, isDir bool, size int64, err error)

GetFileInfo returns the information about the file or directory at 'filePath'.

func (*GCDStorage) IsCacheNeeded 

func (storage *GCDStorage) IsCacheNeeded() bool

If a local snapshot cache is needed for the storage to avoid downloading/uploading chunks too often when managing snapshots.

func (*GCDStorage) IsFastListing 

func (storage *GCDStorage) IsFastListing() bool

If the storage supports fast listing of files names.

func (*GCDStorage) IsMoveFileImplemented 

func (storage *GCDStorage) IsMoveFileImplemented() bool

If the 'MoveFile' method is implemented.

func (*GCDStorage) IsStrongConsistent 

func (storage *GCDStorage) IsStrongConsistent() bool

If the storage can guarantee strong consistency.

func (*GCDStorage) ListFiles 

func (storage *GCDStorage) ListFiles(threadIndex int, dir string) ([]string, []int64, error)

ListFiles return the list of files and subdirectories under 'dir' (non-recursively)

func (*GCDStorage) MoveFile 

func (storage *GCDStorage) MoveFile(threadIndex int, from string, to string) (err error)

MoveFile renames the file.

func (*GCDStorage) UploadFile 

func (storage *GCDStorage) UploadFile(threadIndex int, filePath string, content []byte) (err error)

UploadFile writes 'content' to the file at 'filePath'.

type GCSConfig 

type GCSConfig struct {
	ClientID     string          `json:"client_id"`
	ClientSecret string          `json:"client_secret"`
	Endpoint     oauth2.Endpoint `json:"end_point"`
	Token        oauth2.Token    `json:"token"`
}

type GCSStorage 

type GCSStorage struct {
	StorageBase

	TestMode bool
	// contains filtered or unexported fields
}

func CreateGCSStorage 

func CreateGCSStorage(tokenFile string, bucketName string, storageDir string, threads int) (storage *GCSStorage, err error)

CreateGCSStorage creates a GCD storage object.

func (*GCSStorage) CreateDirectory 

func (storage *GCSStorage) CreateDirectory(threadIndex int, dir string) (err error)

CreateDirectory creates a new directory.

func (*GCSStorage) DeleteFile 

func (storage *GCSStorage) DeleteFile(threadIndex int, filePath string) (err error)

DeleteFile deletes the file or directory at 'filePath'.

func (*GCSStorage) DownloadFile 

func (storage *GCSStorage) DownloadFile(threadIndex int, filePath string, chunk *Chunk) (err error)

DownloadFile reads the file at 'filePath' into the chunk.

func (*GCSStorage) EnableTestMode 

func (storage *GCSStorage) EnableTestMode()

Enable the test mode.

func (*GCSStorage) GetFileInfo 

func (storage *GCSStorage) GetFileInfo(threadIndex int, filePath string) (exist bool, isDir bool, size int64, err error)

GetFileInfo returns the information about the file or directory at 'filePath'.

func (*GCSStorage) IsCacheNeeded 

func (storage *GCSStorage) IsCacheNeeded() bool

If a local snapshot cache is needed for the storage to avoid downloading/uploading chunks too often when managing snapshots.

func (*GCSStorage) IsFastListing 

func (storage *GCSStorage) IsFastListing() bool

If the storage supports fast listing of files names.

func (*GCSStorage) IsMoveFileImplemented 

func (storage *GCSStorage) IsMoveFileImplemented() bool

If the 'MoveFile' method is implemented.

func (*GCSStorage) IsStrongConsistent 

func (storage *GCSStorage) IsStrongConsistent() bool

If the storage can guarantee strong consistency.

func (*GCSStorage) ListFiles 

func (storage *GCSStorage) ListFiles(threadIndex int, dir string) ([]string, []int64, error)

ListFiles return the list of files and subdirectories under 'dir' (non-recursively)

func (*GCSStorage) MoveFile 

func (storage *GCSStorage) MoveFile(threadIndex int, from string, to string) (err error)

MoveFile renames the file.

func (*GCSStorage) UploadFile 

func (storage *GCSStorage) UploadFile(threadIndex int, filePath string, content []byte) (err error)

UploadFile writes 'content' to the file at 'filePath'.

type HubicClient 

type HubicClient struct {
	HTTPClient *http.Client

	TokenFile string
	Token     *oauth2.Token
	TokenLock *sync.Mutex

	Credential     HubicCredential
	CredentialLock *sync.Mutex

	TestMode bool
}

func NewHubicClient 

func NewHubicClient(tokenFile string) (*HubicClient, error)

func (*HubicClient) CreateDirectory 

func (client *HubicClient) CreateDirectory(path string) error

func (*HubicClient) DeleteFile 

func (client *HubicClient) DeleteFile(path string) error

func (*HubicClient) DownloadFile 

func (client *HubicClient) DownloadFile(path string) (io.ReadCloser, int64, error)

func (*HubicClient) GetCredential 

func (client *HubicClient) GetCredential() (err error)

func (*HubicClient) GetFileInfo 

func (client *HubicClient) GetFileInfo(path string) (bool, bool, int64, error)

func (*HubicClient) ListEntries 

func (client *HubicClient) ListEntries(path string) ([]HubicEntry, error)

func (*HubicClient) MoveFile 

func (client *HubicClient) MoveFile(from string, to string) error

func (*HubicClient) RefreshToken 

func (client *HubicClient) RefreshToken(force bool) (err error)

func (*HubicClient) UploadFile 

func (client *HubicClient) UploadFile(path string, content []byte, rateLimit int) (err error)

type HubicCredential 

type HubicCredential struct {
	Token    string
	Endpoint string
	Expires  time.Time
}

type HubicEntry 

type HubicEntry struct {
	Name   string `json:"name"`
	Size   int64  `json:"bytes"`
	Type   string `json:"content_type"`
	Subdir string `json:"subdir"`
}

type HubicError 

type HubicError struct {
	Status  int
	Message string
}

func (HubicError) Error 

func (err HubicError) Error() string

type HubicStorage 

type HubicStorage struct {
	StorageBase
	// contains filtered or unexported fields
}

func CreateHubicStorage 

func CreateHubicStorage(tokenFile string, storagePath string, threads int) (storage *HubicStorage, err error)

CreateHubicStorage creates an Hubic storage object.

func (*HubicStorage) CreateDirectory 

func (storage *HubicStorage) CreateDirectory(threadIndex int, dir string) (err error)

CreateDirectory creates a new directory.

func (*HubicStorage) DeleteFile 

func (storage *HubicStorage) DeleteFile(threadIndex int, filePath string) (err error)

DeleteFile deletes the file or directory at 'filePath'.

func (*HubicStorage) DownloadFile 

func (storage *HubicStorage) DownloadFile(threadIndex int, filePath string, chunk *Chunk) (err error)

DownloadFile reads the file at 'filePath' into the chunk.

func (*HubicStorage) EnableTestMode 

func (storage *HubicStorage) EnableTestMode()

Enable the test mode.

func (*HubicStorage) GetFileInfo 

func (storage *HubicStorage) GetFileInfo(threadIndex int, filePath string) (exist bool, isDir bool, size int64, err error)

GetFileInfo returns the information about the file or directory at 'filePath'.

func (*HubicStorage) IsCacheNeeded 

func (storage *HubicStorage) IsCacheNeeded() bool

If a local snapshot cache is needed for the storage to avoid downloading/uploading chunks too often when managing snapshots.

func (*HubicStorage) IsFastListing 

func (storage *HubicStorage) IsFastListing() bool

If the storage supports fast listing of files names.

func (*HubicStorage) IsMoveFileImplemented 

func (storage *HubicStorage) IsMoveFileImplemented() bool

If the 'MoveFile' method is implemented.

func (*HubicStorage) IsStrongConsistent 

func (storage *HubicStorage) IsStrongConsistent() bool

If the storage can guarantee strong consistency.

func (*HubicStorage) ListFiles 

func (storage *HubicStorage) ListFiles(threadIndex int, dir string) ([]string, []int64, error)

ListFiles return the list of files and subdirectories under 'dir' (non-recursively)

func (*HubicStorage) MoveFile 

func (storage *HubicStorage) MoveFile(threadIndex int, from string, to string) (err error)

MoveFile renames the file.

func (*HubicStorage) UploadFile 

func (storage *HubicStorage) UploadFile(threadIndex int, filePath string, content []byte) (err error)

UploadFile writes 'content' to the file at 'filePath'.

type IncompleteSnapshot 

type IncompleteSnapshot struct {
	Files        []*Entry
	ChunkHashes  []string
	ChunkLengths []int
}

This is the struct used to save/load incomplete snapshots

type ListBucketOutput 

type ListBucketOutput struct {
	AccountID  string
	BucketID   string
	BucketName string
	BucketType string
}

type Logger 

type Logger struct {
	// contains filtered or unexported fields
}

Set up logging for libraries that Duplicacy depends on. They can call 'log.Printf("[ID] message")' to produce logs in Duplicacy's format

func (*Logger) Write 

func (logger *Logger) Write(line []byte) (n int, err error)

type OneDriveClient 

type OneDriveClient struct {
	HTTPClient *http.Client

	TokenFile string
	Token     *oauth2.Token
	TokenLock *sync.Mutex

	IsConnected bool
	TestMode    bool

	IsBusiness      bool
	RefreshTokenURL string
	APIURL          string
}

func NewOneDriveClient 

func NewOneDriveClient(tokenFile string, isBusiness bool) (*OneDriveClient, error)

func (*OneDriveClient) CreateDirectory 

func (client *OneDriveClient) CreateDirectory(path string, name string) error

func (*OneDriveClient) CreateUploadSession 

func (client *OneDriveClient) CreateUploadSession(path string) (uploadURL string, err error)

func (*OneDriveClient) DeleteFile 

func (client *OneDriveClient) DeleteFile(path string) error

func (*OneDriveClient) DownloadFile 

func (client *OneDriveClient) DownloadFile(path string) (io.ReadCloser, int64, error)

func (*OneDriveClient) GetFileInfo 

func (client *OneDriveClient) GetFileInfo(path string) (string, bool, int64, error)

func (*OneDriveClient) ListEntries 

func (client *OneDriveClient) ListEntries(path string) ([]OneDriveEntry, error)

func (*OneDriveClient) MoveFile 

func (client *OneDriveClient) MoveFile(path string, parent string) error

func (*OneDriveClient) RefreshToken 

func (client *OneDriveClient) RefreshToken(force bool) (err error)

func (*OneDriveClient) UploadFile 

func (client *OneDriveClient) UploadFile(path string, content []byte, rateLimit int) (err error)

func (*OneDriveClient) UploadFileSession 

func (client *OneDriveClient) UploadFileSession(uploadURL string, content []byte, rateLimit int) (err error)

type OneDriveEntry 

type OneDriveEntry struct {
	ID     string
	Name   string
	Folder map[string]interface{}
	Size   int64
}

type OneDriveError 

type OneDriveError struct {
	Status  int
	Message string
}

func (OneDriveError) Error 

func (err OneDriveError) Error() string

type OneDriveErrorResponse 

type OneDriveErrorResponse struct {
	Error OneDriveError `json:"error"`
}

type OneDriveListEntriesOutput 

type OneDriveListEntriesOutput struct {
	Entries  []OneDriveEntry `json:"value"`
	NextLink string          `json:"@odata.nextLink"`
}

type OneDriveStorage 

type OneDriveStorage struct {
	StorageBase
	// contains filtered or unexported fields
}

func CreateOneDriveStorage 

func CreateOneDriveStorage(tokenFile string, isBusiness bool, storagePath string, threads int) (storage *OneDriveStorage, err error)

CreateOneDriveStorage creates an OneDrive storage object.

func (*OneDriveStorage) CreateDirectory 

func (storage *OneDriveStorage) CreateDirectory(threadIndex int, dir string) (err error)

CreateDirectory creates a new directory.

func (*OneDriveStorage) DeleteFile 

func (storage *OneDriveStorage) DeleteFile(threadIndex int, filePath string) (err error)

DeleteFile deletes the file or directory at 'filePath'.

func (*OneDriveStorage) DownloadFile 

func (storage *OneDriveStorage) DownloadFile(threadIndex int, filePath string, chunk *Chunk) (err error)

DownloadFile reads the file at 'filePath' into the chunk.

func (*OneDriveStorage) EnableTestMode 

func (storage *OneDriveStorage) EnableTestMode()

Enable the test mode.

func (*OneDriveStorage) GetFileInfo 

func (storage *OneDriveStorage) GetFileInfo(threadIndex int, filePath string) (exist bool, isDir bool, size int64, err error)

GetFileInfo returns the information about the file or directory at 'filePath'.

func (*OneDriveStorage) IsCacheNeeded 

func (storage *OneDriveStorage) IsCacheNeeded() bool

If a local snapshot cache is needed for the storage to avoid downloading/uploading chunks too often when managing snapshots.

func (*OneDriveStorage) IsFastListing 

func (storage *OneDriveStorage) IsFastListing() bool

If the storage supports fast listing of files names.

func (*OneDriveStorage) IsMoveFileImplemented 

func (storage *OneDriveStorage) IsMoveFileImplemented() bool

If the 'MoveFile' method is implemented.

func (*OneDriveStorage) IsStrongConsistent 

func (storage *OneDriveStorage) IsStrongConsistent() bool

If the storage can guarantee strong consistency.

func (*OneDriveStorage) ListFiles 

func (storage *OneDriveStorage) ListFiles(threadIndex int, dir string) ([]string, []int64, error)

ListFiles return the list of files and subdirectories under 'dir' (non-recursively)

func (*OneDriveStorage) MoveFile 

func (storage *OneDriveStorage) MoveFile(threadIndex int, from string, to string) (err error)

MoveFile renames the file.

func (*OneDriveStorage) UploadFile 

func (storage *OneDriveStorage) UploadFile(threadIndex int, filePath string, content []byte) (err error)

UploadFile writes 'content' to the file at 'filePath'.

type Preference 

type Preference struct {
	Name              string            `json:"name"`
	SnapshotID        string            `json:"id"`
	RepositoryPath    string            `json:"repository"`
	StorageURL        string            `json:"storage"`
	Encrypted         bool              `json:"encrypted"`
	BackupProhibited  bool              `json:"no_backup"`
	RestoreProhibited bool              `json:"no_restore"`
	DoNotSavePassword bool              `json:"no_save_password"`
	NobackupFile      string            `json:"nobackup_file"`
	Keys              map[string]string `json:"keys"`
	FiltersFile       string            `json:"filters"`
}

Preference stores options for each storage.

func FindPreference 

func FindPreference(name string) *Preference

func (*Preference) Equal 

func (preference *Preference) Equal(other *Preference) bool

type RateLimitedReader 

type RateLimitedReader struct {
	Content   []byte
	Rate      float64
	Next      int
	StartTime time.Time
}

func CreateRateLimitedReader 

func CreateRateLimitedReader(content []byte, rate int) *RateLimitedReader

func (*RateLimitedReader) Length 

func (reader *RateLimitedReader) Length() int64

func (*RateLimitedReader) Read 

func (reader *RateLimitedReader) Read(p []byte) (n int, err error)

func (*RateLimitedReader) Reset 

func (reader *RateLimitedReader) Reset()

func (*RateLimitedReader) Seek 

func (reader *RateLimitedReader) Seek(offset int64, whence int) (int64, error)

type S3CStorage 

type S3CStorage struct {
	StorageBase
	// contains filtered or unexported fields
}

S3CStorage is a storage backend for s3 compatible storages that require V2 Signing.

func CreateS3CStorage 

func CreateS3CStorage(regionName string, endpoint string, bucketName string, storageDir string,
	accessKey string, secretKey string, threads int) (storage *S3CStorage, err error)

CreateS3CStorage creates a amazon s3 storage object.

func (*S3CStorage) CreateDirectory 

func (storage *S3CStorage) CreateDirectory(threadIndex int, dir string) (err error)

CreateDirectory creates a new directory.

func (*S3CStorage) DeleteFile 

func (storage *S3CStorage) DeleteFile(threadIndex int, filePath string) (err error)

DeleteFile deletes the file or directory at 'filePath'.

func (*S3CStorage) DownloadFile 

func (storage *S3CStorage) DownloadFile(threadIndex int, filePath string, chunk *Chunk) (err error)

DownloadFile reads the file at 'filePath' into the chunk.

func (*S3CStorage) EnableTestMode 

func (storage *S3CStorage) EnableTestMode()

Enable the test mode.

func (*S3CStorage) GetFileInfo 

func (storage *S3CStorage) GetFileInfo(threadIndex int, filePath string) (exist bool, isDir bool, size int64, err error)

GetFileInfo returns the information about the file or directory at 'filePath'.

func (*S3CStorage) IsCacheNeeded 

func (storage *S3CStorage) IsCacheNeeded() bool

If a local snapshot cache is needed for the storage to avoid downloading/uploading chunks too often when managing snapshots.

func (*S3CStorage) IsFastListing 

func (storage *S3CStorage) IsFastListing() bool

If the storage supports fast listing of files names.

func (*S3CStorage) IsMoveFileImplemented 

func (storage *S3CStorage) IsMoveFileImplemented() bool

If the 'MoveFile' method is implemented.

func (*S3CStorage) IsStrongConsistent 

func (storage *S3CStorage) IsStrongConsistent() bool

If the storage can guarantee strong consistency.

func (*S3CStorage) ListFiles 

func (storage *S3CStorage) ListFiles(threadIndex int, dir string) (files []string, sizes []int64, err error)

ListFiles return the list of files and subdirectories under 'dir' (non-recursively)

func (*S3CStorage) MoveFile 

func (storage *S3CStorage) MoveFile(threadIndex int, from string, to string) (err error)

MoveFile renames the file.

func (*S3CStorage) UploadFile 

func (storage *S3CStorage) UploadFile(threadIndex int, filePath string, content []byte) (err error)

UploadFile writes 'content' to the file at 'filePath'.

type S3Storage 

type S3Storage struct {
	StorageBase
	// contains filtered or unexported fields
}

func CreateS3Storage 

func CreateS3Storage(regionName string, endpoint string, bucketName string, storageDir string,
	accessKey string, secretKey string, threads int,
	isSSLSupported bool, isMinioCompatible bool) (storage *S3Storage, err error)

CreateS3Storage creates a amazon s3 storage object.

func (*S3Storage) CreateDirectory 

func (storage *S3Storage) CreateDirectory(threadIndex int, dir string) (err error)

CreateDirectory creates a new directory.

func (*S3Storage) DeleteFile 

func (storage *S3Storage) DeleteFile(threadIndex int, filePath string) (err error)

DeleteFile deletes the file or directory at 'filePath'.

func (*S3Storage) DownloadFile 

func (storage *S3Storage) DownloadFile(threadIndex int, filePath string, chunk *Chunk) (err error)

DownloadFile reads the file at 'filePath' into the chunk.

func (*S3Storage) EnableTestMode 

func (storage *S3Storage) EnableTestMode()

Enable the test mode.

func (*S3Storage) GetFileInfo 

func (storage *S3Storage) GetFileInfo(threadIndex int, filePath string) (exist bool, isDir bool, size int64, err error)

GetFileInfo returns the information about the file or directory at 'filePath'.

func (*S3Storage) IsCacheNeeded 

func (storage *S3Storage) IsCacheNeeded() bool

If a local snapshot cache is needed for the storage to avoid downloading/uploading chunks too often when managing snapshots.

func (*S3Storage) IsFastListing 

func (storage *S3Storage) IsFastListing() bool

If the storage supports fast listing of files names.

func (*S3Storage) IsMoveFileImplemented 

func (storage *S3Storage) IsMoveFileImplemented() bool

If the 'MoveFile' method is implemented.

func (*S3Storage) IsStrongConsistent 

func (storage *S3Storage) IsStrongConsistent() bool

If the storage can guarantee strong consistency.

func (*S3Storage) ListFiles 

func (storage *S3Storage) ListFiles(threadIndex int, dir string) (files []string, sizes []int64, err error)

ListFiles return the list of files and subdirectories under 'dir' (non-recursively)

func (*S3Storage) MoveFile 

func (storage *S3Storage) MoveFile(threadIndex int, from string, to string) (err error)

MoveFile renames the file.

func (*S3Storage) UploadFile 

func (storage *S3Storage) UploadFile(threadIndex int, filePath string, content []byte) (err error)

UploadFile writes 'content' to the file at 'filePath'.

type SFTPStorage 

type SFTPStorage struct {
	StorageBase
	// contains filtered or unexported fields
}

func CreateSFTPStorage 

func CreateSFTPStorage(compatibilityMode bool, server string, port int, username string, storageDir string, minimumNesting int,
	authMethods []ssh.AuthMethod,
	hostKeyCallback func(hostname string, remote net.Addr,
		key ssh.PublicKey) error, threads int) (storage *SFTPStorage, err error)

func CreateSFTPStorageWithPassword 

func CreateSFTPStorageWithPassword(server string, port int, username string, storageDir string,
	minimumNesting int, password string, threads int) (storage *SFTPStorage, err error)

func (*SFTPStorage) CreateDirectory 

func (storage *SFTPStorage) CreateDirectory(threadIndex int, dirPath string) (err error)

CreateDirectory creates a new directory.

func (*SFTPStorage) DeleteFile 

func (storage *SFTPStorage) DeleteFile(threadIndex int, filePath string) (err error)

DeleteFile deletes the file or directory at 'filePath'.

func (*SFTPStorage) DownloadFile 

func (storage *SFTPStorage) DownloadFile(threadIndex int, filePath string, chunk *Chunk) (err error)

DownloadFile reads the file at 'filePath' into the chunk.

func (*SFTPStorage) EnableTestMode 

func (storage *SFTPStorage) EnableTestMode()

Enable the test mode.

func (*SFTPStorage) GetFileInfo 

func (storage *SFTPStorage) GetFileInfo(threadIndex int, filePath string) (exist bool, isDir bool, size int64, err error)

GetFileInfo returns the information about the file or directory at 'filePath'.

func (*SFTPStorage) IsCacheNeeded 

func (storage *SFTPStorage) IsCacheNeeded() bool

If a local snapshot cache is needed for the storage to avoid downloading/uploading chunks too often when managing snapshots.

func (*SFTPStorage) IsFastListing 

func (storage *SFTPStorage) IsFastListing() bool

If the storage supports fast listing of files names.

func (*SFTPStorage) IsMoveFileImplemented 

func (storage *SFTPStorage) IsMoveFileImplemented() bool

If the 'MoveFile' method is implemented.

func (*SFTPStorage) IsStrongConsistent 

func (storage *SFTPStorage) IsStrongConsistent() bool

If the storage can guarantee strong consistency.

func (*SFTPStorage) ListFiles 

func (storage *SFTPStorage) ListFiles(threadIndex int, dirPath string) (files []string, sizes []int64, err error)

ListFiles return the list of files and subdirectories under 'file' (non-recursively)

func (*SFTPStorage) MoveFile 

func (storage *SFTPStorage) MoveFile(threadIndex int, from string, to string) (err error)

MoveFile renames the file.

func (*SFTPStorage) UploadFile 

func (storage *SFTPStorage) UploadFile(threadIndex int, filePath string, content []byte) (err error)

UploadFile writes 'content' to the file at 'filePath'.

type Snapshot 

type Snapshot struct {
	ID            string // the snapshot id; must be different for different repositories
	Revision      int    // the revision number
	Options       string // options used to create this snapshot (some not included)
	Tag           string // user-assigned tag
	StartTime     int64  // at what time the snapshot was created
	EndTime       int64  // at what time the snapshot was done
	FileSize      int64  // total file size
	NumberOfFiles int64  // number of files

	// A sequence of chunks whose aggregated content is the json representation of 'Files'.
	FileSequence []string

	// A sequence of chunks whose aggregated content is the json representation of 'ChunkHashes'.
	ChunkSequence []string

	// A sequence of chunks whose aggregated content is the json representation of 'ChunkLengths'.
	LengthSequence []string

	Files []*Entry // list of files and subdirectories

	ChunkHashes  []string // a sequence of chunks representing the file content
	ChunkLengths []int    // the length of each chunk

	Flag bool // used to mark certain snapshots for deletion or copy
	// contains filtered or unexported fields
}

Snapshot represents a backup of the repository.

func CreateEmptySnapshot 

func CreateEmptySnapshot(id string) (snapshto *Snapshot)

CreateEmptySnapshot creates an empty snapshot.

func CreateSnapshotFromDescription 

func CreateSnapshotFromDescription(description []byte) (snapshot *Snapshot, err error)

CreateSnapshotFromDescription creates a snapshot from json decription.

func CreateSnapshotFromDirectory 

func CreateSnapshotFromDirectory(id string, top string, nobackupFile string, filtersFile string) (snapshot *Snapshot, skippedDirectories []string,
	skippedFiles []string, err error)

CreateSnapshotFromDirectory creates a snapshot from the local directory 'top'. Only 'Files' will be constructed, while 'ChunkHashes' and 'ChunkLengths' can only be populated after uploading.

func LoadIncompleteSnapshot 

func LoadIncompleteSnapshot() (snapshot *Snapshot)

LoadIncompleteSnapshot loads the incomplete snapshot if it exists

func (*Snapshot) ClearChunks 

func (snapshot *Snapshot) ClearChunks()

ClearChunks removes loaded chunks from memory

func (*Snapshot) LoadChunks 

func (snapshot *Snapshot) LoadChunks(description []byte) (err error)

LoadChunks construct 'ChunkHashes' from the json description.

func (*Snapshot) LoadLengths 

func (snapshot *Snapshot) LoadLengths(description []byte) (err error)

LoadLengths construct 'ChunkLengths' from the json description.

func (*Snapshot) MarshalJSON 

func (snapshot *Snapshot) MarshalJSON() ([]byte, error)

MarshalJSON creates a json representation of the snapshot.

func (*Snapshot) MarshalSequence 

func (snapshot *Snapshot) MarshalSequence(sequenceType string) ([]byte, error)

MarshalSequence creates a json represetion for the specified chunk sequence.

func (*Snapshot) SetSequence 

func (snapshot *Snapshot) SetSequence(sequenceType string, sequence []string)

SetSequence assign a chunk sequence to the specified field.

type SnapshotManager 

type SnapshotManager struct {
	// contains filtered or unexported fields
}

SnapshotManager is mainly responsible for downloading, and deleting snapshots.

func CreateSnapshotManager 

func CreateSnapshotManager(config *Config, storage Storage) *SnapshotManager

CreateSnapshotManager creates a snapshot manager

func (*SnapshotManager) CheckSnapshot 

func (manager *SnapshotManager) CheckSnapshot(snapshot *Snapshot) (err error)

CheckSnapshot performs sanity checks on the given snapshot.

func (*SnapshotManager) CheckSnapshots 

func (manager *SnapshotManager) CheckSnapshots(snapshotID string, revisionsToCheck []int, tag string, showStatistics bool, showTabular bool,
	checkFiles bool, checkChunks, searchFossils bool, resurrect bool, threads int) bool

ListSnapshots shows the information about a snapshot.

func (*SnapshotManager) CleanSnapshotCache 

func (manager *SnapshotManager) CleanSnapshotCache(latestSnapshot *Snapshot, allSnapshots map[string][]*Snapshot) bool

CleanSnapshotCache removes all files not referenced by the specified 'snapshot' in the snapshot cache.

func (*SnapshotManager) ClearSnapshotContents 

func (manager *SnapshotManager) ClearSnapshotContents(snapshot *Snapshot)

ClearSnapshotContents removes contents loaded by DownloadSnapshotContents

func (*SnapshotManager) ConvertSequence 

func (manager *SnapshotManager) ConvertSequence(sequence []string) (result []string)

ConvertSequence converts a sequence of chunk hashes into a sequence of chunk ids.

func (*SnapshotManager) CreateChunkDownloader 

func (manager *SnapshotManager) CreateChunkDownloader()

func (*SnapshotManager) Diff 

func (manager *SnapshotManager) Diff(top string, snapshotID string, revisions []int,
	filePath string, compareByHash bool, nobackupFile string, filtersFile string) bool

Diff compares two snapshots, or two revision of a file if the file argument is given.

func (*SnapshotManager) DownloadFile 

func (manager *SnapshotManager) DownloadFile(path string, derivationKey string) (content []byte)

DownloadFile downloads a non-chunk file from the storage. The only non-chunk files in the current implementation are snapshot files.

func (*SnapshotManager) DownloadSequence 

func (manager *SnapshotManager) DownloadSequence(sequence []string) (content []byte)

DownloadSequence returns the content represented by a sequence of chunks.

func (*SnapshotManager) DownloadSnapshot 

func (manager *SnapshotManager) DownloadSnapshot(snapshotID string, revision int) *Snapshot

DownloadSnapshot downloads the specified snapshot.

func (*SnapshotManager) DownloadSnapshotContents 

func (manager *SnapshotManager) DownloadSnapshotContents(snapshot *Snapshot, patterns []string, attributesNeeded bool) bool

DownloadSnapshotContents loads all chunk sequences in a snapshot. A snapshot, when just created, only contains some metadata and theree sequence representing files, chunk hashes, and chunk lengths. This function must be called for the actual content of the snapshot to be usable.

func (*SnapshotManager) DownloadSnapshotFileSequence 

func (manager *SnapshotManager) DownloadSnapshotFileSequence(snapshot *Snapshot, patterns []string, attributesNeeded bool) bool

func (*SnapshotManager) DownloadSnapshotSequence 

func (manager *SnapshotManager) DownloadSnapshotSequence(snapshot *Snapshot, sequenceType string) bool

DownloadSnapshotSequence downloads the content represented by a sequence of chunks, and then unmarshal the content using the specified 'loadFunction'. It purpose is to decode the chunk sequences representing chunk hashes or chunk lengths in a snapshot.

func (*SnapshotManager) FindFile 

func (manager *SnapshotManager) FindFile(snapshot *Snapshot, filePath string, suppressError bool) *Entry

FindFile returns the file entry that has the given file name.

func (*SnapshotManager) GetSnapshotChunkHashes 

func (manager *SnapshotManager) GetSnapshotChunkHashes(snapshot *Snapshot, chunkHashes *map[string]bool, chunkIDs map[string]bool)

GetSnapshotChunkHashes has an option to retrieve chunk hashes in addition to chunk ids.

func (*SnapshotManager) GetSnapshotChunks 

func (manager *SnapshotManager) GetSnapshotChunks(snapshot *Snapshot, keepChunkHashes bool) (chunks []string)

GetSnapshotChunks returns all chunks referenced by a given snapshot. If keepChunkHashes is true, snapshot.ChunkHashes will be populated.

func (*SnapshotManager) ListAllFiles 

func (manager *SnapshotManager) ListAllFiles(storage Storage, top string) (allFiles []string, allSizes []int64)

ListAllFiles return all files and subdirectories in the subtree of the 'top' directory in the specified 'storage'.

func (*SnapshotManager) ListSnapshotIDs 

func (manager *SnapshotManager) ListSnapshotIDs() (snapshotIDs []string, err error)

ListSnapshotIDs returns all snapshot ids.

func (*SnapshotManager) ListSnapshotRevisions 

func (manager *SnapshotManager) ListSnapshotRevisions(snapshotID string) (revisions []int, err error)

ListSnapshotRevisions returns the list of all revisions given a snapshot id.

func (*SnapshotManager) ListSnapshots 

func (manager *SnapshotManager) ListSnapshots(snapshotID string, revisionsToList []int, tag string,
	showFiles bool, showChunks bool) int

ListSnapshots shows the information about a snapshot.

func (*SnapshotManager) PrintFile 

func (manager *SnapshotManager) PrintFile(snapshotID string, revision int, path string) bool

PrintFile prints the specified file or the snapshot to stdout.

func (*SnapshotManager) PrintSnapshot 

func (manager *SnapshotManager) PrintSnapshot(snapshot *Snapshot) bool

PrintSnapshot prints the snapshot in the json format (with chunk hasher converted into chunk ids)

func (*SnapshotManager) PruneSnapshots 

func (manager *SnapshotManager) PruneSnapshots(selfID string, snapshotID string, revisionsToBeDeleted []int,
	tags []string, retentions []string,
	exhaustive bool, exclusive bool, ignoredIDs []string,
	dryRun bool, deleteOnly bool, collectOnly bool, threads int) bool

PruneSnapshots deletes snapshots by revisions, tags, or a retention policy. The main idea is two-step fossil collection.

  1. Delete snapshots specified by revision, retention policy, with a tag. Find any resulting unreferenced chunks, and mark them as fossils (by renaming). After that, create a fossil collection file containing fossils collected during current run, and temporary files encountered. Also in the file is the latest revision for each snapshot id. Save this file to a local directory.
  1. On next run, check if there is any new revision for each snapshot. Or if the lastest revision is too old, for instance, more than 7 days. This step is to identify snapshots that were being created while step 1 is in progress. For each fossil reference by any of these snapshots, move them back to the normal chunk directory.

Note that a snapshot being created when step 2 is in progress may reference a fossil. To avoid this problem, never remove the lastest revision (unless exclusive is true), and only cache chunks referenced by the lastest revision.

func (*SnapshotManager) RetrieveFile 

func (manager *SnapshotManager) RetrieveFile(snapshot *Snapshot, file *Entry, output func([]byte)) bool

RetrieveFile retrieves the file in the specified snapshot.

func (*SnapshotManager) ShowHistory 

func (manager *SnapshotManager) ShowHistory(top string, snapshotID string, revisions []int,
	filePath string, showLocalHash bool) bool

ShowHistory shows how a file changes over different revisions.

func (*SnapshotManager) ShowStatistics 

func (manager *SnapshotManager) ShowStatistics(snapshotMap map[string][]*Snapshot, chunkSizeMap map[string]int64, chunkUniqueMap map[string]bool,
	chunkSnapshotMap map[string]int)

Print snapshot and revision statistics

func (*SnapshotManager) ShowStatisticsTabular 

func (manager *SnapshotManager) ShowStatisticsTabular(snapshotMap map[string][]*Snapshot, chunkSizeMap map[string]int64, chunkUniqueMap map[string]bool,
	chunkSnapshotMap map[string]int)

Print snapshot and revision statistics in tabular format

func (*SnapshotManager) UploadFile 

func (manager *SnapshotManager) UploadFile(path string, derivationKey string, content []byte) bool

UploadFile uploads a non-chunk file from the storage.

func (*SnapshotManager) VerifySnapshot 

func (manager *SnapshotManager) VerifySnapshot(snapshot *Snapshot) bool

VerifySnapshot verifies that every file in the snapshot has the correct hash. It does this by downloading chunks and computing the whole file hash for each file.

type Storage 

type Storage interface {
	// ListFiles return the list of files and subdirectories under 'dir'.  A subdirectories returned must have a trailing '/', with
	// a size of 0.  If 'dir' is 'snapshots', only subdirectories will be returned.  If 'dir' is 'snapshots/repository_id', then only
	// files will be returned.  If 'dir' is 'chunks', the implementation can return the list either recusively or non-recusively.
	ListFiles(threadIndex int, dir string) (files []string, sizes []int64, err error)

	// DeleteFile deletes the file or directory at 'filePath'.
	DeleteFile(threadIndex int, filePath string) (err error)

	// MoveFile renames the file.
	MoveFile(threadIndex int, from string, to string) (err error)

	// CreateDirectory creates a new directory.
	CreateDirectory(threadIndex int, dir string) (err error)

	// GetFileInfo returns the information about the file or directory at 'filePath'.
	GetFileInfo(threadIndex int, filePath string) (exist bool, isDir bool, size int64, err error)

	// FindChunk finds the chunk with the specified id.  If 'isFossil' is true, it will search for chunk files with
	// the suffix '.fsl'.
	FindChunk(threadIndex int, chunkID string, isFossil bool) (filePath string, exist bool, size int64, err error)

	// DownloadFile reads the file at 'filePath' into the chunk.
	DownloadFile(threadIndex int, filePath string, chunk *Chunk) (err error)

	// UploadFile writes 'content' to the file at 'filePath'.
	UploadFile(threadIndex int, filePath string, content []byte) (err error)

	// SetNestingLevels sets up the chunk nesting structure.
	SetNestingLevels(config *Config)

	// If a local snapshot cache is needed for the storage to avoid downloading/uploading chunks too often when
	// managing snapshots.
	IsCacheNeeded() bool

	// If the 'MoveFile' method is implemented.
	IsMoveFileImplemented() bool

	// If the storage can guarantee strong consistency.
	IsStrongConsistent() bool

	// If the storage supports fast listing of files names.
	IsFastListing() bool

	// Enable the test mode.
	EnableTestMode()

	// Set the maximum transfer speeds.
	SetRateLimits(downloadRateLimit int, uploadRateLimit int)
}

func CreateStorage 

func CreateStorage(preference Preference, resetPassword bool, threads int) (storage Storage)

CreateStorage creates a storage object based on the provide storage URL.

type StorageBase 

type StorageBase struct {
	DownloadRateLimit int // Maximum download rate (bytes/seconds)
	UploadRateLimit   int // Maximum upload reate (bytes/seconds)

	DerivedStorage Storage // Used as the pointer to the derived storage class
	// contains filtered or unexported fields
}

StorageBase is the base struct from which all storages are derived from

func (*StorageBase) FindChunk 

func (storage *StorageBase) FindChunk(threadIndex int, chunkID string, isFossil bool) (filePath string, exist bool, size int64, err error)

FindChunk finds the chunk with the specified id at the levels one by one as specified by 'readLevels'.

func (*StorageBase) SetDefaultNestingLevels 

func (storage *StorageBase) SetDefaultNestingLevels(readLevels []int, writeLevel int)

SetDefaultNestingLevels sets the default read and write levels. This is usually called by derived storages to set the levels with old values so that storages initialized by earlier versions will continue to work.

func (*StorageBase) SetNestingLevels 

func (storage *StorageBase) SetNestingLevels(config *Config)

SetNestingLevels sets the new read and write levels (normally both at 1) if the 'config' file has the 'fixed-nesting' key, or if a file named 'nesting' exists on the storage.

func (*StorageBase) SetRateLimits 

func (storage *StorageBase) SetRateLimits(downloadRateLimit int, uploadRateLimit int)

SetRateLimits sets the maximum download and upload rates

type SwiftStorage 

type SwiftStorage struct {
	StorageBase
	// contains filtered or unexported fields
}

func CreateSwiftStorage 

func CreateSwiftStorage(storageURL string, key string, threads int) (storage *SwiftStorage, err error)

CreateSwiftStorage creates an OpenStack Swift storage object. storageURL is in the form of `user@authURL/container/path?arg1=value1&arg2=value2“

func (*SwiftStorage) CreateDirectory 

func (storage *SwiftStorage) CreateDirectory(threadIndex int, dir string) (err error)

CreateDirectory creates a new directory.

func (*SwiftStorage) DeleteFile 

func (storage *SwiftStorage) DeleteFile(threadIndex int, filePath string) (err error)

DeleteFile deletes the file or directory at 'filePath'.

func (*SwiftStorage) DownloadFile 

func (storage *SwiftStorage) DownloadFile(threadIndex int, filePath string, chunk *Chunk) (err error)

DownloadFile reads the file at 'filePath' into the chunk.

func (*SwiftStorage) EnableTestMode 

func (storage *SwiftStorage) EnableTestMode()

Enable the test mode.

func (*SwiftStorage) GetFileInfo 

func (storage *SwiftStorage) GetFileInfo(threadIndex int, filePath string) (exist bool, isDir bool, size int64, err error)

GetFileInfo returns the information about the file or directory at 'filePath'.

func (*SwiftStorage) IsCacheNeeded 

func (storage *SwiftStorage) IsCacheNeeded() bool

If a local snapshot cache is needed for the storage to avoid downloading/uploading chunks too often when managing snapshots.

func (*SwiftStorage) IsFastListing 

func (storage *SwiftStorage) IsFastListing() bool

If the storage supports fast listing of files names.

func (*SwiftStorage) IsMoveFileImplemented 

func (storage *SwiftStorage) IsMoveFileImplemented() bool

If the 'MoveFile' method is implemented.

func (*SwiftStorage) IsStrongConsistent 

func (storage *SwiftStorage) IsStrongConsistent() bool

If the storage can guarantee strong consistency.

func (*SwiftStorage) ListFiles 

func (storage *SwiftStorage) ListFiles(threadIndex int, dir string) (files []string, sizes []int64, err error)

ListFiles return the list of files and subdirectories under 'dir' (non-recursively)

func (*SwiftStorage) MoveFile 

func (storage *SwiftStorage) MoveFile(threadIndex int, from string, to string) (err error)

MoveFile renames the file.

func (*SwiftStorage) UploadFile 

func (storage *SwiftStorage) UploadFile(threadIndex int, filePath string, content []byte) (err error)

UploadFile writes 'content' to the file at 'filePath'.

type WasabiStorage 

type WasabiStorage struct {
	StorageBase
	// contains filtered or unexported fields
}

func CreateWasabiStorage 

func CreateWasabiStorage(
	regionName string, endpoint string,
	bucketName string, storageDir string,
	accessKey string, secretKey string,
	threads int,
) (storage *WasabiStorage, err error)

func (*WasabiStorage) CreateDirectory 

func (storage *WasabiStorage) CreateDirectory(
	threadIndex int, dir string,
) (err error)

func (*WasabiStorage) DeleteFile 

func (storage *WasabiStorage) DeleteFile(
	threadIndex int, filePath string,
) (err error)

func (*WasabiStorage) DownloadFile 

func (storage *WasabiStorage) DownloadFile(
	threadIndex int, filePath string, chunk *Chunk,
) (err error)

func (*WasabiStorage) EnableTestMode 

func (storage *WasabiStorage) EnableTestMode()

func (*WasabiStorage) GetFileInfo 

func (storage *WasabiStorage) GetFileInfo(
	threadIndex int, filePath string,
) (exist bool, isDir bool, size int64, err error)

func (*WasabiStorage) IsCacheNeeded 

func (storage *WasabiStorage) IsCacheNeeded() bool

func (*WasabiStorage) IsFastListing 

func (storage *WasabiStorage) IsFastListing() bool

func (*WasabiStorage) IsMoveFileImplemented 

func (storage *WasabiStorage) IsMoveFileImplemented() bool

func (*WasabiStorage) IsStrongConsistent 

func (storage *WasabiStorage) IsStrongConsistent() bool

func (*WasabiStorage) ListFiles 

func (storage *WasabiStorage) ListFiles(
	threadIndex int, dir string,
) (files []string, sizes []int64, err error)

func (*WasabiStorage) MoveFile 

func (storage *WasabiStorage) MoveFile(threadIndex int, from string, to string) (err error)

This is a lightweight implementation of a call to Wasabi for a rename. It's designed to get the job done with as few dependencies on other packages as possible rather than being somethng general-purpose and reusable.

func (*WasabiStorage) UploadFile 

func (storage *WasabiStorage) UploadFile(
	threadIndex int, filePath string, content []byte,
) (err error)

type WebDAVMultiStatus 

type WebDAVMultiStatus struct {
	Responses []WebDAVResponse `xml:"response"`
}

type WebDAVProp 

type WebDAVProp struct {
	PropList []WebDAVPropValue `xml:",any"`
}

type WebDAVPropStat 

type WebDAVPropStat struct {
	Prop *WebDAVProp `xml:"prop"`
}

type WebDAVPropValue 

type WebDAVPropValue struct {
	XMLName xml.Name `xml:""`
	Value   string   `xml:",innerxml"`
}

type WebDAVProperties 

type WebDAVProperties map[string]string

type WebDAVResponse 

type WebDAVResponse struct {
	Href     string          `xml:"href"`
	PropStat *WebDAVPropStat `xml:"propstat"`
}

type WebDAVStorage 

type WebDAVStorage struct {
	StorageBase
	// contains filtered or unexported fields
}

func CreateWebDAVStorage 

func CreateWebDAVStorage(host string, port int, username string, password string, storageDir string, useHTTP bool, threads int) (storage *WebDAVStorage, err error)

func (*WebDAVStorage) CreateDirectory 

func (storage *WebDAVStorage) CreateDirectory(threadIndex int, dir string) (err error)

CreateDirectory creates a new directory.

func (*WebDAVStorage) DeleteFile 

func (storage *WebDAVStorage) DeleteFile(threadIndex int, filePath string) (err error)

DeleteFile deletes the file or directory at 'filePath'.

func (*WebDAVStorage) DownloadFile 

func (storage *WebDAVStorage) DownloadFile(threadIndex int, filePath string, chunk *Chunk) (err error)

DownloadFile reads the file at 'filePath' into the chunk.

func (*WebDAVStorage) EnableTestMode 

func (storage *WebDAVStorage) EnableTestMode()

Enable the test mode.

func (*WebDAVStorage) GetFileInfo 

func (storage *WebDAVStorage) GetFileInfo(threadIndex int, filePath string) (exist bool, isDir bool, size int64, err error)

GetFileInfo returns the information about the file or directory at 'filePath'.

func (*WebDAVStorage) IsCacheNeeded 

func (storage *WebDAVStorage) IsCacheNeeded() bool

If a local snapshot cache is needed for the storage to avoid downloading/uploading chunks too often when managing snapshots.

func (*WebDAVStorage) IsFastListing 

func (storage *WebDAVStorage) IsFastListing() bool

If the storage supports fast listing of files names.

func (*WebDAVStorage) IsMoveFileImplemented 

func (storage *WebDAVStorage) IsMoveFileImplemented() bool

If the 'MoveFile' method is implemented.

func (*WebDAVStorage) IsStrongConsistent 

func (storage *WebDAVStorage) IsStrongConsistent() bool

If the storage can guarantee strong consistency.

func (*WebDAVStorage) ListFiles 

func (storage *WebDAVStorage) ListFiles(threadIndex int, dir string) (files []string, sizes []int64, err error)

ListFiles return the list of files and subdirectories under 'dir'. A subdirectories returned must have a trailing '/', with a size of 0. If 'dir' is 'snapshots', only subdirectories will be returned. If 'dir' is 'snapshots/repository_id', then only files will be returned. If 'dir' is 'chunks', the implementation can return the list either recusively or non-recusively.

func (*WebDAVStorage) MoveFile 

func (storage *WebDAVStorage) MoveFile(threadIndex int, from string, to string) (err error)

MoveFile renames the file.

func (*WebDAVStorage) UploadFile 

func (storage *WebDAVStorage) UploadFile(threadIndex int, filePath string, content []byte) (err error)

UploadFile writes 'content' to the file at 'filePath'.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.