DataStore

The DataStore interface in comby serves as the foundation for managing binary and structured data. It defines a set of operations for storing, retrieving, copying, deleting, and managing data objects within a storage system. Designed for extensibility, the DataStore interface enables developers to implement custom storage backends while maintaining compatibility with the comby framework.

Interface

The DataStore interface provides methods for initialization, data operations (retrieval, storage, copying, listing, and deletion), metadata utilities (e.g., totals, store info), and state management (e.g., closing connections and resetting the store). This ensures flexibility and efficient data handling tailored to specific configurations and use cases.

type DataStore interface {
	// Init initializes the data store with the provided options.
	Init(ctx context.Context, opts ...DataStoreOption) error

	// Get retrieves a data object by its bucket and object name.
	Get(ctx context.Context, opts ...DataStoreGetOption) (*DataModel, error)

	// Set adds or updates a data object with the specified bucket and object name.
	Set(ctx context.Context, opts ...DataStoreSetOption) error

	// Copy copies a data object from a source to a destination within the store.
	Copy(ctx context.Context, opts ...DataStoreCopyOption) error

	// List retrieves a list of data objects based on the provided options.
	List(ctx context.Context, opts ...DataStoreListOption) ([]*DataModel, int64, error)

	// Delete removes a data object by its bucket and object name.
	Delete(ctx context.Context, opts ...DataStoreDeleteOption) error

	// Total returns the total number of data objects in the store.
	Total(ctx context.Context) int64

	// Close closes the data store connection.
	Close(ctx context.Context) error

	// Options returns the configuration options of the data store.
	Options() DataStoreOptions

	// String returns a string representation of the data store.
	String() string

	// Info provides detailed information about the data store.
	Info(ctx context.Context) (*DataInfoModel, error)

	// Reset clears all data objects from the store.
	Reset(ctx context.Context) error
}

Model

The DataModel represents a data object stored within the data store. It includes metadata such as the bucket name, object name, size, last modification timestamp, and the actual data content.

type DataModel struct {
	BucketName   string `json:"bucketName"`   // Name of the bucket containing the object.
	ObjectName   string `json:"objectName"`   // Name of the object within the bucket.
	Size         int64  `json:"size"`         // Size of the data object in bytes.
	LastModified int64  `json:"lastModified"` // Timestamp of the last modification.
	Data         []byte `json:"-"`            // Actual data content (not serialized).
}

Options and Methods

Init

The Init method sets up the data store with specific configurations, such as custom endpoints or logging preferences.

Options	Description
DataStoreListOptionWithTenantUuid(tenantUuid string)	Sets the tenant UUID for filtering the data store.
DataStoreOptionWithReadOnly(readOnly bool)	Specifies whether the store is read-only.
DataStoreOptionWithLogLevel(logLevel string)	Sets the log level for the data store.
DataStoreOptionWithAttribute(keys string, value any)	Sets custom attributes for the data store.

Get

The Get method fetches individual data objects identified by their bucket and object names. Returns cache entry of model DataModel.

Options	Description
DataStoreGetOptionWithBucketName(bucketName string)	Specifies the bucket name of the data object.
DataStoreGetOptionWithObjectName(objectName string)	Specifies the object name of the data object.
DataStoreGetOptionWithAttribute(keys string, value any)	Sets custom attributes for the Get option.

Set

The Set method allows for adding new data objects or updating existing ones.

Options	Description
DataStoreSetOptionWithBucketName(bucketName string)	Specifies the bucket name of the data object.
DataStoreSetOptionWithObjectName(objectName string)	Specifies the object name of the data object.
DataStoreSetOptionWithContentType(contentType string)	Specifies the content type of the data object.
DataStoreSetOptionWithData(data []byte)	Specifies the data content of the data object.
DataStoreSetOptionWithAttribute(keys string, value any)	Sets custom attributes for the Set object.

Copy

The Copy method facilitates duplicating data between buckets or renaming within the store.

Options	Description
DataStoreCopyOptionWithSrcBucketName(bucketName string)	Specifies the source bucket name.
DataStoreCopyOptionWithSrcObjectName(objectName string)	Specifies the source object name.
DataStoreCopyOptionWithDstBucketName(bucketName string)	Specifies the destination bucket name.
DataStoreCopyOptionWithDstObjectName(objectName string)	Specifies the destination object name.
DataStoreCopyOptionWithAttributes(attributes *AttributeMap)	Sets custom attributes for the copied object.
DataStoreCopyOptionWithAttribute(keys string, value any)	Sets custom attributes for the Copy object.

List

The List method retrieves multiple data objects, supporting pagination and sorting. Returns list of cache entries of model DataModel.

Options	Description
DataStoreListOptionOffset(offset int64)	Sets the offset for pagination.
DataStoreListOptionLimit(limit int64)	Sets the limit for pagination.
DataStoreListOptionOrderBy(orderBy string)	Sets the field to order the data objects by.
DataStoreListOptionAscending(ascending bool)	Sets the sorting order (ascending or descending).
DataStoreListOptionWithAttribute(keys string, value any)	Sets custom attributes for the List object.

Delete

The Delete method removes specific data objects from the store.

Options	Description
DataStoreDeleteOptionWithBucketName(bucketName string)	Specifies the bucket name of the data
object.
DataStoreDeleteOptionWithObjectName(objectName string)	Specifies the object name of the data object.
DataStoreDeleteOptionWithAttribute(keys string, value any)	Sets custom attributes for the Delete object.

Total

Provides the total count of stored objects.

Close

Closes connections to the data store gracefully.

Options

Returns the configuration options of the data store.

String

Returns a string representation of the data store.

Info

Returns detailed metadata about the store, such as storage type, number of objects, and connection details. Details are returned in the following structure:

type DataStoreInfoModel struct {
	StoreType        string `json:"storeType"`        // Type of the data store.
	ConnectionInfo   string `json:"connectionInfo"`   // Connection details of the data store.
	LastUpdateTime   int64  `json:"lastUpdateTime"`   // Timestamp of the last update.
	NumBuckets       int64  `json:"numBuckets"`       // Number of buckets in the store.
	NumObjects       int64  `json:"numObjects"`       // Number of objects in the store.
	TotalSizeInBytes int64  `json:"totalSizeInBytes"` // Total size of all objects in bytes.
	InfoText         string `json:"infoText"`         // Additional information about the store.
}

Reset

Clears all data objects from the store, resetting it to an empty state.

Extensibility and Implementation

comby provides the following default implementations of the DataStore interface:

• In-Memory Store: Useful for testing and development, offering fast and transient data storage. Used by default if no specific store is configured.
• FileSystem Store: A production-ready implementation leveraging the local file system for data storage.
• MinIO Store: A cloud-native implementation using MinIO as the storage backend. Ideal for scalable and distributed data storage in cloud environments. Link: https://github.com/gradientzero/comby-store-minio

Users can implement the DataStore interface to integrate with alternative storage systems, such as:

• Amazon S3
• Google Cloud Storage
• Azure Blob Storage
• Distributed file systems (e.g., HDFS, Ceph)

To create a custom data store, implement all methods in the interface, adhering to the specified contracts for behavior and parameters.