Hallucination Attacks in Fastapi with Firestore

Hallucination attacks in a Fastapi service that uses Firestore occur when an LLM or AI-generated response fabricates information that appears authoritative but is not grounded in the stored data. Because Fastapi often serves as a backend for AI features—such as summarizing documents, answering queries over datasets, or suggesting metadata—developers may pass Firestore document snapshots or query results into a language model without first validating or constraining the model’s scope. When the model lacks sufficient context or when the input data contains ambiguities, it may invent plausible-sounding fields, IDs, or relationships that do not exist in Firestore.

In this combination, the vulnerability is amplified by two factors common to Fastapi and Firestore. First, Fastapi’s flexible request handling and automatic JSON serialization can inadvertently expose raw Firestore document structures, including internal fields such as __name__ or metadata that an attacker might leverage to infer collection hierarchies. Second, Firestore’s schemaless design allows documents to vary widely within a collection; if the application does not enforce strict schemas before sending data to the model, the LLM may misinterpret missing keys as nulls or hallucinate entire subcollections that were never written.

Attackers can exploit this through prompt injection techniques that manipulate the data sent to the model. For example, by modifying query parameters or request bodies, an adversary can cause the Fastapi endpoint to retrieve a subset of Firestore documents that omit key identifiers (such as tenant IDs or access scopes). The LLM, receiving incomplete context, may then hallucinate access to records it should not see or assert relationships that violate business rules. This aligns with BOLA/IDOR patterns where object-level authorization is assumed but not enforced. Because middleBrick’s LLM/AI Security checks specifically test for system prompt leakage, active prompt injection, and unsafe consumption of model outputs, it can detect when a Fastapi endpoint exposes LLM endpoints without proper guardrails, allowing fabricated responses to leak sensitive assumptions or internal logic.

Another vector involves Firestore’s indexing and query behavior. If a Fastapi route builds dynamic queries using user-supplied filters and passes the resulting document references directly to an LLM for natural language generation, the model may hallucinate document contents that do not match the indexed data. This is especially risky when using aggregation-like patterns where counts or summaries are inferred by the model rather than computed server-side. The risk is not merely informational; an attacker could chain hallucinated outputs with other API endpoints to escalate privileges or infer data indirectly. middleBrick’s checks for unsafe consumption, inventory management, and property authorization are designed to surface these gaps by correlating runtime behavior with OpenAPI specifications and Firestore-like data exposure patterns.

Because Firestore does not inherently enforce output constraints, the onus is on the Fastapi layer to validate and sanitize before LLM interaction. Hallucination attacks in this stack therefore represent a breakdown in data integrity and context enforcement, where the model’s generative nature is exploited through poorly bounded queries and unverified assumptions about document structure. Detecting these issues requires correlating API behavior with LLM response anomalies, a capability emphasized in middleBrick’s framework-agnostic findings, which map to OWASP API Top 10 and related compliance guidance.

To mitigate hallucination attacks when using Fastapi with Firestore, enforce strict data validation and context boundaries before passing information to any LLM. The following patterns assume you are using the google-cloud-firestore library and Pydantic models for request and response validation.

1. Explicitly scope queries and validate document existence

Always anchor queries to a known tenant or user ID and verify that documents exist before sending data to the model. Do not rely on the model to infer missing records.

from fastapi import FastAPI, Depends, HTTPException
from google.cloud import firestore
from pydantic import BaseModel
from typing import List

app = FastAPI()
db = firestore.Client()

class QueryRequest(BaseModel):
    tenant_id: str
    document_ids: List[str]

@app.post("/generate-summary")
def generate_summary(request: QueryRequest):
    collection_ref = db.collection("tenants")
    tenant_ref = collection_ref.document(request.tenant_id)
    if not tenant_ref.get().exists:
        raise HTTPException(status_code=404, detail="Tenant not found")
    
    docs = []
    for doc_id in request.document_ids:
        doc_ref = tenant_ref.collection("documents").document(doc_id)
        doc = doc_ref.get()
        if not doc.exists:
            raise HTTPException(status_code=404, detail=f"Document {doc_id} not found")
        docs.append(doc.to_dict())
    
    # Only pass validated data to the model
    return {"documents": docs}

2. Enforce schema consistency and reject dynamic fields

Define a strict Pydantic schema for Firestore documents before LLM consumption to prevent the model from inventing fields. Reject any document that does not conform.

from pydantic import ValidationError
from typing import Dict, Any

class FirestoreDocument(BaseModel):
    title: str
    content: str
    created_at: str  # ISO format
    tags: List[str]

def validate_documents(raw_docs: List[Dict[str, Any]]) -> List[FirestoreDocument]:
    validated = []
    for raw in raw_docs:
        # Reject extra fields to avoid hallucinated metadata exploitation
        if set(raw.keys()) != {"title", "content", "created_at", "tags"}:
            raise ValueError("Document schema mismatch")
        validated.append(FirestoreDocument(**raw))
    return validated

3. Use server-side aggregation instead of model inference

Compute counts, summaries, or relationships in Firestore or in your Fastapi layer rather than asking the LLM to infer them, which reduces hallucination surface.

from google.cloud import firestore

def get_tenant_document_counts(tenant_id: str) -> int:
    collection_ref = db.collection("tenants").document(tenant_id).collection("documents")
    # Firestore count aggregation (client-side estimate or server-side via batch)
    docs = collection_ref.stream()
    return sum(1 for _ in docs)

4. Isolate LLM inputs and outputs with strict allowlists

When passing data to an LLM endpoint, use allowlisted fields only and avoid forwarding raw Firestore metadata. Treat LLM responses as untrusted and validate them before any downstream use.

def sanitize_for_llm(document: FirestoreDocument) -> Dict[str, str]:
    # Only expose safe fields to the model
    return {
        "title": document.title,
        "content": document.content
    }

5. Apply continuous scanning for unsafe consumption patterns

Use tools that inspect how Firestore data flows into language models. MiddleBrick’s LLM/AI Security checks can identify system prompt leakage, unsafe consumption, and missing authorization context in your API routes, helping to catch hallucination risks early.