gokit

A modular Go toolkit for building production services.

gokit provides a shared foundation across Go services — config, logging, resilience, observability, dependency injection, and infrastructure adapters — so teams can focus on business logic instead of reinventing plumbing.

Architecture

gokit uses a multi-module layout:

• Core module (github.com/kbukum/gokit) — lightweight, zero heavy dependencies. Covers config, logging, errors, DI, resilience, and abstractions.
• Sub-modules (github.com/kbukum/gokit/{name}) — each has its own go.mod and brings in heavier dependencies (Gin, GORM, Kafka, gRPC, etc.) only when you need them.

Import the core for foundational utilities. Add sub-modules à la carte for infrastructure.

Compatibility Matrix

Go Version	Module Version
1.25+	v0.1.2+

Module Map

Core Packages

Package	Import	Description
errors	gokit/errors	Structured errors with codes, HTTP status mapping, and RFC 7807 support
config	gokit/config	Base configuration with environment-specific settings and defaults
logger	gokit/logger	Structured logging via zerolog with context injection and component tagging
util	gokit/util	Generic slice, map, pointer, and functional utilities
version	gokit/version	Build version info — git commit, branch, build time, dirty state
encryption	gokit/encryption	AES-256-GCM encryption and decryption for sensitive data
validation	gokit/validation	Struct tag and programmatic validation with field-level error collection
di	gokit/di	Dependency injection container with lazy/eager init, retry, and circuit breaker
resilience	gokit/resilience	Circuit breaker, retry with backoff, bulkhead isolation, rate limiting
observability	gokit/observability	OpenTelemetry tracing, metrics, and health checking
sse	gokit/sse	Server-sent events broadcasting with per-client channels
provider	gokit/provider	Generic provider framework with metadata, state management, middleware, sink combinators, and runtime checks
pipeline	gokit/pipeline	Pull-based data pipeline with Throttle, Batch, Debounce, and Window operators
dag	gokit/dag	DAG execution engine — dependency-ordered orchestration with batch, streaming, and cascade modes
media	gokit/media	Media type detection from content bytes — video, audio, image, text format identification
security	gokit/security	Security utilities
component	gokit/component	Lifecycle interface for infrastructure components (start/stop/health)
bootstrap	gokit/bootstrap	Application startup orchestration and graceful shutdown

Sub-Modules

Module	Import	Description
auth	gokit/auth	Authentication — JWT tokens, OIDC verification, password hashing, token validation interfaces
authz	gokit/authz	Authorization — permission checking, wildcard pattern matching (zero external deps)
database	gokit/database	PostgreSQL via GORM — pooling, migrations, health checks, slow query logging
redis	gokit/redis	go-redis client wrapper with pooling, health checks, TypedStore, and JSON operations
httpclient	gokit/httpclient	HTTP client with resilience patterns, retry, and circuit breaking
kafka	gokit/kafka	Kafka producer/consumer with TLS/SASL, configurable transport
storage	gokit/storage	Object storage abstraction — local filesystem and S3-compatible backends
server	gokit/server	HTTP server with Gin, HTTP/2, middleware stack, handler mounting
grpc	gokit/grpc	gRPC client config — TLS, keepalive, message size, connection pooling
discovery	gokit/discovery	Service discovery with Consul and static provider support
connect	gokit/connect	Connect-Go RPC registration over HTTP/1.1 with standardized errors
process	gokit/process	Subprocess execution with context cancellation and signal handling
workload	gokit/workload	Workload execution on Docker and Kubernetes backends
llm	gokit/llm	LLM chat completion abstraction — dialect-based provider mapping, streaming, structured output
bench	gokit/bench	Evaluation benchmarking framework — datasets, evaluators, metrics, reports, comparison
bench/viz	gokit/bench/viz	SVG visualization generation — ROC curves, confusion matrices, calibration plots
bench/storage	gokit/bench/storage	Bench storage adapter — bridges bench.RunStorage with gokit/storage backends

Quick Start

Install the core

go get github.com/kbukum/gokit@latest

Add a sub-module

go get github.com/kbukum/gokit/server@latest
go get github.com/kbukum/gokit/database@latest

Usage Examples

Config + Logger

package main

import (
    "github.com/kbukum/gokit/config"
    "github.com/kbukum/gokit/logger"
)

type ServiceConfig struct {
    config.ServiceConfig `yaml:",inline" mapstructure:",squash"`
    Port int `yaml:"port"`
}

func main() {
    cfg := &ServiceConfig{}
    if err := config.LoadConfig("my-service", cfg,
        config.WithConfigFile("./config.yml"),
        config.WithEnvFile(".env"),
    ); err != nil {
        panic(err)
    }
    cfg.ApplyDefaults()

    log := logger.New(&logger.Config{
        Level:  "info",
        Format: "console",
    }, cfg.Name)

    log.Info("service configured", map[string]interface{}{
        "env": cfg.Environment,
    })
}

HTTP Server with Middleware

import "github.com/kbukum/gokit/server"

srvCfg := &server.Config{Host: "0.0.0.0", Port: 8080}
srvCfg.ApplyDefaults()

srv := server.New(srvCfg, log)
srv.ApplyDefaults("my-service", healthChecker)

srv.GinEngine().GET("/api/items", itemsHandler)

srv.Start(ctx)
defer srv.Stop(ctx)

Provider Pattern

import "github.com/kbukum/gokit/provider"

// Define a domain provider using the interaction pattern
type DiarizationProvider = provider.RequestResponse[AudioInput, []Segment]

// Use the manager for runtime selection
reg := provider.NewRegistry[DiarizationProvider]()
mgr := provider.NewManager(reg, &provider.HealthCheckSelector[DiarizationProvider]{})
p, _ := mgr.Get(ctx)
result, err := p.Execute(ctx, audioInput)

Sink Composition

import "github.com/kbukum/gokit/provider"

// Wrap a plain function as a Sink
kafkaSink := provider.NewSinkFunc("kafka", func(ctx context.Context, event Event) error {
    return producer.Publish(ctx, topic, event)
})

// Fan out to multiple sinks in parallel
sink := provider.FanOutSink("multi",
    kafkaSink,
    provider.AdaptSink(analyticsSink, "adapt", toAnalyticsEvent),
    provider.TapSink(loggingSink, func(ctx context.Context, e Event) {
        metrics.RecordEvent(e.Type)
    }),
)

// Compose sink middleware
wrapped := provider.ChainSink(withLogging, withMetrics)(sink)
wrapped.Send(ctx, event) // dispatches to all sinks with logging + metrics

Subprocess Execution

import "github.com/kbukum/gokit/process"

result, err := process.Run(ctx, process.Command{
    Binary: "python", Args: []string{"diarize.py", "audio.wav"},
})
fmt.Println(string(result.Stdout))

Bootstrap Lifecycle

import "github.com/kbukum/gokit/bootstrap"

app := bootstrap.NewApp("my-service", "1.0.0",
    bootstrap.WithLogger(log),
    bootstrap.WithGracefulTimeout(15 * time.Second),
)

app.RegisterComponent(db)    // component.Component
app.RegisterComponent(cache) // component.Component

app.OnConfigure(func(ctx context.Context, app *bootstrap.App) error {
    // All components started — set up routes, handlers, business logic
    return nil
})

// Run: Init → Start → Configure → Ready → wait for signal → Stop
if err := app.Run(ctx); err != nil {
    log.Fatal("app failed", map[string]interface{}{"error": err})
}

Module Details

Each module has its own documentation. Refer to the package-level Go docs or source:

Group	Packages	Focus
Foundation	errors, config, logger, version	Configuration, logging, error handling
Utilities	util, encryption, validation	Common helpers and data validation
Architecture	di, provider, component, bootstrap	DI, lifecycle management, provider pattern
Auth & Authz	auth, authz	Authentication (JWT, OIDC, password) and authorization (permissions)
Resilience	resilience, observability	Fault tolerance, tracing, metrics
Data	pipeline, dag, sse, media	Pull-based pipelines, DAG orchestration, server-sent events, media detection
Infrastructure	database, redis, kafka, storage	Data stores and messaging
Networking	httpclient	HTTP client with resilience
Transport	server, grpc, connect, discovery	HTTP, gRPC, service discovery
Execution	process, workload	Subprocess and container workload execution
AI	llm	LLM chat completion, structured output, explanation generation
Evaluation	bench, bench/viz, bench/storage	Provider benchmarking, metrics, visualizations, result storage

Multi-Module Versioning

Core and sub-modules version independently. Each sub-module has its own go.mod and release tags:

v0.5.0              ← core module
server/v0.3.2       ← server sub-module
database/v0.4.1     ← database sub-module

This means:

• Upgrading gokit/server does not force an upgrade of gokit/database.
• Core can ship breaking changes without touching sub-modules (and vice versa).
• Each module follows semver on its own timeline.

Development

make check    # build + vet + test (all modules)
make test     # run tests with -race across all modules
make lint     # golangci-lint across all modules
make fmt      # gofmt -s
make tidy     # go mod tidy for core + all sub-modules

Contributing

See CONTRIBUTING.md for development workflow, coding standards, and how to submit pull requests.

We follow the Contributor Covenant Code of Conduct.

License

MIT — Copyright (c) 2024 kbukum