Memory Leak in Fiber with Dynamodb

A memory leak in a Fiber-based service that uses the AWS SDK for DynamoDB typically arises when responses from DynamoDB are not fully consumed or closed, or when SDK clients and request-scoped objects are reused without proper cleanup. In Go, this can manifest as continually growing RSS/RAM usage under load, often tied to unclosed response bodies, lingering goroutines, or accumulation of unreferenced objects held by retryers, middleware, or custom HTTP transports.

With DynamoDB, common patterns that contribute to leaks include:

• Not closing the response body from DynamoDB API calls (e.g., Query, Scan, GetItem), which holds network buffers and prevents garbage collection.
• Retaining references to request parameters or model objects across requests, especially when using global or package-level DynamoDB clients combined with mutable state.
• Improper use of context: passing a background context where cancellation is needed, or failing to cancel request-scoped contexts, which can delay cleanup of in-flight responses and associated buffers.
• Middleware or logging that captures request or response objects without trimming large payloads, causing retained memory growth under sustained traffic.

Because middleBrick scans unauthenticated attack surfaces and checks properties like Input Validation and Unsafe Consumption, it can surface related anomalies (for example, unexpectedly large or missing response size limits) that hint at resource handling issues, though it reports findings without fixing them. The scan’s runtime checks do not instrument Go runtime internals, but the API behavior observed—such as missing pagination limits or unbounded response consumption—can indicate patterns that, in a Go service, correlate with memory retention issues.

To illustrate, consider a Fiber handler that repeatedly calls Query on a DynamoDB table and streams results into a growing slice without closing the iterator or consuming pages correctly:

import (
    "context"
    "github.com/gofiber/fiber/v2"
    "github.com/aws/aws-sdk-go-v2/service/dynamodb"
    "github.com/aws/aws-sdk-go-v2/service/dynamodb/types"
)

type Item struct {
    ID   string `json:"id"`
    Data string `json:"data"`
}

func ScanTable(client *dynamodb.Client, tableName string) ([]Item, error) {
    var results []Item
    input := &dynamodb.ScanInput{
        TableName: aws.String(tableName),
    }
    paginator := dynamodb.NewScanPaginator(client, input)
    for paginator.HasMorePages() {
        page, err := paginator.NextPage(context.TODO())
        if err != nil {
            return nil, err
        }
        for _, item := range page.Items {
            results = append(results, Item{
                ID:   aws.ToString(item["id"]),
                Data: aws.ToString(item["data"]),
            })
        }
    }
    return results, nil
}

func handler(c *fiber.Ctx) error {
    items, err := ScanTable(dynamoClient, "widgets")
    if err != nil {
        return c.Status(fiber.StatusInternalServerError).JSON(fiber.Map{"error": err.Error()})
    }
    return c.JSON(items)
}

In this example, potential contributors to memory growth include:

• The paginator retains internal state and HTTP response bodies until NextPage completes; if errors occur mid-scan or contexts are not canceled, underlying buffers may not be released promptly.
• The items slice grows unbounded with each request; if the handler is invoked concurrently, per-request allocations accumulate quickly.
• Using a global DynamoDB client is fine, but if the client carries retryer configurations or HTTP transports with large buffers, and those are coupled with long-lived contexts or middleware that logs full responses, memory can rise steadily under load.

middleBrick’s checks for Rate Limiting and Data Exposure can highlight missing constraints on response sizes or missing backpressure signals, which, when combined with unbounded in-memory aggregation in Fiber handlers, signals a pattern that in Go can lead to memory exhaustion. Remediation focuses on strict resource handling, context discipline, and avoiding retention of large payloads beyond the request scope.

Apply the following patterns to prevent memory retention when using DynamoDB in a Fiber service. The goal is to ensure response bodies are closed, contexts are canceled promptly, pagination does not accumulate unbounded result sets, and shared clients are safely reused without leaking request-specific state.

1) Always consume and close response bodies; use context with cancellation and timeouts; avoid unbounded accumulation.

import (
    "context"
    "time"
    "github.com/gofiber/fiber/v2"
    "github.com/aws/aws-sdk-go-v2/service/dynamodb"
    "github.com/aws/aws-sdk-go-v2/aws"
)

func QueryWithClose(client *dynamodb.Client, tableName string, limit int64) ([]map[string]interface{}, error) {
    ctx, cancel := context.WithTimeout(context.Background(), 8*time.Second)
    defer cancel() // ensures cleanup if we return early

    input := &dynamodb.QueryInput{
        TableName:                 aws.String(tableName),
        Limit:                     aws.Int32(int32(limit)),
        ReturnConsumedCapacity:    aws.String("NONE"),
    }
    var results []map[string]interface{}
    paginator := dynamodb.NewQueryPaginator(client, input)
    for paginator.HasMorePages() {
        page, err := paginator.NextPage(ctx)
        if err != nil {
            return nil, err
        }
        // trim: avoid retaining the full SDK model; extract only needed fields
        for _, item := range page.Items {
            results = append(results, map[string]interface{}{
                "id":   aws.ToString(item["id"]),
                "data": aws.ToString(item["data"]),
            })
        }
        // Early exit if we have enough data to bound memory growth
        if int64(len(results)) >= limit {
            break
        }
    }
    return results, nil
}

func handler(c *fiber.Ctx) error {
    items, err := QueryWithClose(dynamoClient, "widgets", 100)
    if err != nil {
        return c.Status(fiber.StatusBadRequest).JSON(fiber.Map{"error": err.Error()})
    }
    return c.JSON(items)
}

2) Reuse the DynamoDB client safely across requests, but do not share mutable request state; prefer dependency injection to pass a configured client.

// Global client initialization (safe for concurrent use)
var dynamoClient *dynamodb.Client

func init() {
    // Configure client with reasonable timeouts and retry settings
    cfg, err := config.LoadDefaultConfig(context.TODO(),
        config.WithRegion("us-east-1"),
        config.WithHTTPClient(&http.Client{
            Timeout: 10 * time.Second,
            Transport: &http.Transport{
                MaxIdleConns:        100,
                MaxIdleConnsPerHost: 10,
                IdleConnTimeout:     30 * time.Second,
            },
        }),
    )
    if err != nil {
        log.Fatalf("unable to load SDK config: %v", err)
    }
    dynamoClient = dynamodb.NewFromConfig(cfg)
}

3) For scans used in administrative endpoints, enforce strict limits and stream with cancellation; avoid using Scan in hot paths.

func LimitedScan(client *dynamodb.Client, tableName string, maxItems int64) ([]map[string]interface{}, error) {
    ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
    defer cancel()

    input := &dynamodb.ScanInput{
        TableName:           aws.String(tableName),
        Limit:               aws.Int32(int32(maxItems)),
        ConsistentRead:      aws.Bool(false),
        ReturnConsumedCapacity: aws.String("NONE"),
    }
    var out []map[string]interface{}
    paginator := dynamodb.NewScanPaginator(client, input)
    for paginator.HasMorePages() {
        page, err := paginator.NextPage(ctx)
        if err != nil {
            return nil, err
        }
        for _, item := range page.Items {
            out = append(out, convertItem(item))
            if int64(len(out)) >= maxItems {
                return out, nil
            }
        }
    }
    return out, nil
}

func convertItem(item map[string]types.AttributeValue) map[string]interface{} {
    // Convert selectively; avoid retaining full SDK attribute values longer than necessary
    result := make(map[string]interface{}, len(item))
    for k, v := range item {
        // Simplified conversion for example
        if s, ok := v.(*types.AttributeValueMemberS); ok {
            result[k] = s.Value
        } else if b, ok := v.(*types.AttributeValueMemberBOOL); ok {
            result[k] = b.Value
        } else {
            result[k] = fmt.Sprintf("%v", v)
        }
    }
    return result
}

4) Instrument middleware to avoid retaining large payloads and ensure response body closure; prefer streaming or chunked processing where feasible.

func LoggingMiddleware(next fiber.Handler) fiber.Handler {
    return func(c *fiber.Ctx) error {
        // Limit captured payload size; do not store full response in memory
        return next(c) // let response stream through; avoid appending to large buffers
    }
}

These patterns reduce the risk of unbounded memory growth by ensuring timely release of HTTP response resources, bounding paginated result sets, and safely reusing the DynamoDB client. Combined with middleBrick’s checks for Input Validation and Unsafe Consumption, they help align runtime behavior with safer resource handling expectations.