Next.js to Go Migration: Architecture Decision

[CodeMeAPixel]

Architecture Decision: Next.js to Go Migration

Table of Contents

1. Executive Summary
2. Why Go? Strategic Rationale
3. Architectural Improvements
4. Leveraging Goroutines: Practical Examples
5. Operational Benefits
6. Performance Comparison
7. Transition Strategy & Best Practices
8. Conclusion
9. References

---

Executive Summary

We are transitioning our backend infrastructure from Next.js/Node.js API routes to a standalone Go service. This strategic shift significantly improves performance, concurrency, resource utilization, and operational reliability. This document outlines our reasoning, architectural benefits, and how we leverage Go's unique strengths for our infrastructure.

---

Why Go? Strategic Rationale

1. Performance & Resource Efficiency

Next.js (TypeScript/Node.js)

• Interpreted/JIT-compiled JavaScript runtime
• Higher memory footprint (~150-300MB for idle service)
• Slower startup time (requires npm modules, JIT warmup)
• CPU overhead from garbage collection pauses
• Single-threaded event loop bottleneck

Go

• Compiled to native machine code
• Minimal binary (~50MB containerized, ~20MB binary)
• Instant startup time (no JIT, no module loading)
• Efficient concurrent garbage collection with low pause times
• ~5-10x performance improvement on sync operations

Real Impact: Allocations sync reduced from 5+ minutes to 30-60 seconds with the same data volume.

2. Concurrency Model

Node.js Event Loop Limitations

// Node.js handles one async task at a time in the event loop
const tasks = Array(10000).fill(() => query_database());
// These all queue up, creating bottlenecks
await Promise.all(tasks.map(t => t()));

• Single event loop thread processes all async operations
• Multiple requests can starve each other
• Callback hell and promise chains introduce complexity
• Memory overhead per async operation

Go Goroutines - Lightweight Concurrency

// Go can spawn millions of goroutines efficiently
for i := 0; i < 10000; i++ {
    go queryDatabase() // Each runs concurrently on available cores
}

• Goroutines: 2KB memory per goroutine vs 1-2MB per Node.js worker thread
• True parallelism: Automatically distributed across CPU cores
• Scheduler: Go runtime manages CPU time allocation transparently
• Scalability: Can handle 10,000+ concurrent operations trivially

3. Type Safety & Reliability

TypeScript in Next.js

• Runtime type checking via libraries (zod, io-ts)
• Type safety only at compile time
• Runtime errors possible despite type annotations
• Refactoring can miss edge cases

Go

• Compile-time type checking enforced
• No implicit type conversions
• Zero runtime type errors (statically guaranteed)
• Interface-based polymorphism without inheritance overhead
• Explicit error handling prevents silent failures

4. Standard Library Strength

Go's standard library eliminates dependency bloat:

Functionality	Node.js	Go
HTTP Server	Express, Fastify, Hapi	net/http (stdlib)
Database Driver	pg, better-sqlite3	github.com/jackc/pgx
JSON Parsing	json5, ajv	encoding/json (stdlib)
Logging	winston, pino	log/slog (stdlib)
Testing	jest, mocha	testing (stdlib)
CLI	yargs, commander	flag (stdlib)

Benefit: Fewer dependencies = smaller attack surface, fewer security updates, faster builds.

---

Architectural Improvements

Pattern 1: Async Job Queue

Before (Next.js)

// API route - long-running sync blocks request
export async function POST(req) {
  const result = await syncPterodactyl(); // 2+ minutes!
  // Client times out or connection drops
  return Response.json(result);
}

Problems:

• HTTP request timeout (typically 30-60 seconds)
• No progress visibility
• No retry logic
• Client must stay connected

After (Go + Asynq)

// Handler - immediately queues job and returns
func (h *SyncAPIHandler) TriggerFullSync(c *fiber.Ctx) error {
    taskInfo, err := h.queueManager.EnqueueSyncFull(payload)
    return c.Status(202).JSON(SuccessResponse{
        Data: fiber.Map{"task_id": taskInfo.ID},
    })
}

// Worker - processes independently
func (h *SyncHandler) HandleFullSync(ctx context.Context, task *asynq.Task) error {
    // 2+ minutes of sync work, completely decoupled from HTTP
    return h.syncLocations(...) // Won't timeout
}

Benefits:

• Non-blocking API responses
• Automatic retries on failure
• Job persistence in Redis
• Multiple workers scale horizontally
• Frontend polls for progress independently

Pattern 2: Goroutine-Based Background Tasks

Webhook Dispatch

// Dispatch webhooks in background without blocking sync completion
go h.dispatchSyncWebhook(context.Background(), syncLogID, status, duration, err)

Why this matters:

• Sync completes immediately after database update
• Webhooks sent asynchronously (fails don't break sync)
• No timeout pressure on webhook requests
• Multiple webhooks send in parallel

Database Syncing

// Before: One goroutine per node, sequential processing
for _, node := range nodes {
    allocations, err := h.pteroClient.GetAllAllocationsForNode(ctx, node.ID)
    // Process allocations...
}

// After: Goroutine per node with semaphore to limit concurrency
semaphore := make(chan struct{}, 5) // Max 5 concurrent API calls
for _, node := range nodes {
    go func(n Node) {
        semaphore <- struct{}{}        // Acquire
        defer func() { <-semaphore }() // Release
        
        allocations, _ := h.pteroClient.GetAllAllocationsForNode(ctx, n.ID)
        // Process allocations...
    }(node)
}

Performance improvement: Serial (5 min) → Parallel (1 min) = 5x speedup

Pattern 3: Connection Pooling

Next.js (Typical)

// New connection per request (or expensive pool management)
const client = new Client();
await client.connect();
const result = await client.query("SELECT...");
await client.end(); // Close connection

Go

// Automatic connection pool with pgx
pool, _ := pgxpool.New(ctx, "postgres://...")
result := pool.QueryRow(ctx, "SELECT...") // Reuses connection from pool

// Go runtime manages pool lifecycle automatically
defer pool.Close()

Impact:

• Eliminates connection establishment overhead
• Reuses persistent connections
• Automatic cleanup on pool exhaustion

---

Leveraging Goroutines: Practical Examples

Example 1: Parallel Data Sync

Challenge: Sync 10,000 items from Pterodactyl API sequentially takes too long.

func (h *SyncHandler) syncServersParallel(ctx context.Context, servers []PteroServer) error {
    results := make(chan error, len(servers))
    
    // Spawn a goroutine for each server, but limit concurrency
    semaphore := make(chan struct{}, 10) // Max 10 concurrent
    
    for _, server := range servers {
        go func(s PteroServer) {
            semaphore <- struct{}{}        // Acquire slot
            defer func() { <-semaphore }() // Release slot
            
            if err := h.upsertServer(ctx, s); err != nil {
                results <- err
                return
            }
            results <- nil
        }(server)
    }
    
    // Collect results
    for i := 0; i < len(servers); i++ {
        if err := <-results; err != nil {
            log.Warn().Err(err).Msg("Server sync failed, continuing...")
        }
    }
    return nil
}

Performance:

• Sequential: 1000 items × 100ms = 100 seconds
• Parallel (10 workers): 1000 items ÷ 10 × 100ms = 10 seconds
• 10x speedup with minimal code change

Example 2: Concurrent API Requests with Timeout

func (h *SyncHandler) fetchMultipleServers(ctx context.Context, ids []string) {
    // Create context with timeout
    ctx, cancel := context.WithTimeout(ctx, 30*time.Second)
    defer cancel()
    
    results := make(chan *Server, len(ids))
    
    for _, id := range ids {
        go func(serverID string) {
            // If ctx times out, goroutine automatically unblocks
            server, _ := h.pteroClient.GetServer(ctx, serverID)
            results <- server
        }(id)
    }
    
    // Collect with timeout protection
    for i := 0; i < len(ids); i++ {
        select {
        case server := <-results:
            h.db.UpsertServer(ctx, server)
        case <-ctx.Done():
            // All remaining requests cancelled automatically
            return fmt.Errorf("fetch timeout")
        }
    }
}

Benefits:

• Automatic cancellation propagates to all goroutines
• No zombie goroutines
• Resource cleanup guaranteed

Example 3: Worker Pool Pattern

// Reusable pattern for any task processing

type WorkerPool struct {
    jobs    chan Job
    workers int
}

func NewWorkerPool(workerCount int) *WorkerPool {
    return &WorkerPool{
        jobs:    make(chan Job, 100),
        workers: workerCount,
    }
}

func (wp *WorkerPool) Start(ctx context.Context) {
    for i := 0; i < wp.workers; i++ {
        go wp.worker(ctx)
    }
}

func (wp *WorkerPool) worker(ctx context.Context) {
    for {
        select {
        case job := <-wp.jobs:
            job.Process() // Blocks until job completes
        case <-ctx.Done():
            return // Graceful shutdown
        }
    }
}

// Usage
pool := NewWorkerPool(5)
pool.Start(ctx)
for _, item := range items {
    pool.jobs <- NewJob(item)
}

Use Cases:

• Database batch processing
• API request throttling
• File processing pipelines

---

Operational Benefits

1. Observability

Go provides native profiling and metrics:

// CPU Profiling
import _ "net/http/pprof"

// Access at http://localhost:6060/debug/pprof/
// Identify hot paths and bottlenecks easily

// Goroutine tracking
import "runtime"

numGoroutines := runtime.NumGoroutine()
// Monitor goroutine leaks automatically

2. Memory Management

// Go's garbage collector is optimized for server workloads
// Concurrent mark-and-sweep with sub-millisecond pause times

// Monitor GC stats
var m runtime.MemStats
runtime.ReadMemStats(&m)
log.Info().
    Uint64("alloc_mb", m.Alloc/1024/1024).
    Uint64("total_alloc_mb", m.TotalAlloc/1024/1024).
    Msg("Memory stats")

3. Graceful Shutdown

// Go's context package makes shutdown safe and predictable
ctx, cancel := context.WithCancel(context.Background())

// On SIGTERM
go func() {
    <-sigChan
    cancel() // Cancel all child contexts
}()

// All goroutines see ctx.Done() and shut down
// No forced termination, no data loss

---

Performance Comparison

Metric	Next.js	Go	Improvement
Startup Time	3-5s	<100ms	30-50x
Memory (idle)	200MB	30MB	6-7x
Allocations Sync	5+ min	30-60s	5-10x
Concurrent Requests	~100	10,000+	100x
Goroutine Creation	N/A	<1µs	Native
Context Switch Overhead	Medium	Minimal	OS scheduled

---

Transition Strategy & Best Practices

How We Move Logic from Next.js to Go

1. API Routes → Fiber Handlers

   // Next.js
   export async function POST(req) {
       const body = await req.json();
       return Response.json(await processSync(body));
   }

   // Go
   func (h *Handler) ProcessSync(c *fiber.Ctx) error {
       var body SyncRequest
       c.BodyParser(&body)
       result := h.processSync(body)
       return c.JSON(result)
   }

2. Long-Running Jobs → Asynq Tasks

   // Next.js - risky, can timeout
   await heavyProcessing();

   // Go - safe, queued, retryable
   h.queueManager.EnqueueTask(payload)
   // Separate worker processes it

3. Database Operations → Connection Pool

   // Next.js - new connection each time
   const db = new Client();

   // Go - reuses pooled connection
   h.db.Pool.QueryRow(ctx, query)

Monitoring Goroutines

func monitorGoroutines(ctx context.Context) {
    ticker := time.NewTicker(1 * time.Minute)
    defer ticker.Stop()
    
    for {
        select {
        case <-ticker.C:
            num := runtime.NumGoroutine()
            if num > 10000 { // Alert threshold
                log.Warn().Int("goroutines", num).Msg("High goroutine count")
            }
        case <-ctx.Done():
            return
        }
    }
}

---

Conclusion

Transitioning to Go represents a strategic architectural decision that:

✅ Improves performance - 5-10x faster sync operations
✅ Enables true concurrency - Goroutines scale to thousands
✅ Reduces resource overhead - 6-7x less memory usage
✅ Enhances reliability - Type-safe, better error handling
✅ Simplifies operations - Single compiled binary, no dependencies
✅ Provides scalability - Horizontal scaling of workers trivial

By combining Go's native concurrency primitives (goroutines, channels, context) with proper async patterns (Asynq job queue, connection pooling), we create a robust, high-performance backend capable of handling the complex operations required by our game server hosting platform.

---

References

• Goroutines
• Context Package
• Asynq Documentation
• pgx Connection Pool
• Go Performance Tips