Llm Data Leakage in Flask with Firestore

When a Flask application integrates Google Cloud Firestore and exposes endpoints that return Firestore documents to an LLM-enabled feature (such as an AI chat or assistant), there is a risk of LLM data leakage. This occurs when sensitive Firestore documents are inadvertently included in prompts or responses, allowing an LLM to reveal confidential data through its outputs.

Flask does not inherently expose Firestore data, but common patterns increase risk. For example, a route that retrieves a user profile document and passes it directly into a system prompt can leak PII or secrets if the LLM response is captured or logged. Consider this pattern:

from flask import Flask, request, jsonify
from google.cloud import firestore

app = Flask(__name__)
db = firestore.Client()

@app.route("/assistant")
def assistant():
    user_id = request.args.get("user_id")
    doc_ref = db.collection("users").document(user_id)
    doc = doc_ref.get()
    if not doc.exists:
        return jsonify({"error": "not found"}), 404
    # Risky: entire document used in a system prompt
    system_prompt = f"You are assisting user {doc.to_dict()}"
    # Assume llm_chat is a helper that sends the prompt to an LLM
    response = llm_chat(system_prompt)
    return jsonify({"response": response})

If the Firestore document contains fields like email, internal_id, or role, and the LLM is misconfigured or prompts are crafted to extract system information, these details may appear in the model’s output. middleBrick’s LLM/AI Security checks detect system prompt leakage by scanning for sensitive patterns in both the prompt construction and LLM responses.

Three dimensions heighten the exposure when using Flask with Firestore and LLMs:

• Prompt construction: Serializing a full Firestore document (e.g., doc.to_dict()) into a system prompt is risky because it may include fields that should remain confidential. Even benign-seeming fields can aid an attacker if combined with other context.
• LLM output handling: If the Flask route returns or logs the LLM’s raw response, and that response includes repeated or inferred sensitive data (PII, API keys, or internal identifiers), the data is effectively leaked through the model.
• Unauthenticated endpoints: If the route is unauthenticated or relies on easily guessable identifiers (e.g., sequential IDs), an attacker can enumerate documents and probe the LLM to exfiltrate data, a pattern middleBrick flags as unauthenticated LLM endpoint exposure alongside potential LLM/AI Security issues like prompt injection and cost exploitation.

To mitigate LLM data leakage with Firestore in Flask, limit what you pass to the LLM and control what the model can return. Apply the principle of least privilege to both Firestore reads and prompt content.

Remediation pattern 1: Select only required fields

Instead of passing the entire document, construct a minimal context object. This reduces the chance that sensitive fields reach the LLM or appear in logs.

from flask import Flask, request, jsonify
from google.cloud import firestore

app = Flask(__name__)
db = firestore.Client()

def get_public_user_summary(user_id: str) -> dict:
    doc_ref = db.collection("users").document(user_id)
    doc = doc_ref.get()
    if not doc.exists:
        return {}
    data = doc.to_dict()
    # Return only non-sensitive fields
    return {
        "display_name": data.get("display_name"),
        "role": data.get("role")
    }

@app.route("/assistant")
def assistant():
    user_id = request.args.get("user_id")
    summary = get_public_user_summary(user_id)
    if not summary:
        return jsonify({"error": "not found"}), 404
    # Safe: only non-sensitive fields in the prompt
    system_prompt = f"You are assisting user {summary.get('display_name')} with role {summary.get('role')}"
    response = llm_chat(system_prompt)
    return jsonify({"response": response})

Remediation pattern 2: Validate and sanitize LLM outputs

Even with restricted prompts, ensure responses are inspected for unintended data. Use output scanning to detect PII or secrets before returning data to the client. If your stack supports post-processing, redact or remove sensitive matches.

import re

def sanitize_llm_output(text: str) -> str:
    # Example: remove potential API keys (simple heuristic)
    text = re.sub(r\"\"\"[aA][pP][iI]_?[kK][eE][yY]\s*[=: ]\s*['\"][a-zA-Z0-9\-_]{20,}['\"]""\", "[REDACTED]", text)
    # Add additional patterns for emails, internal IDs, etc.
    return text

@app.route("/assistant")
def assistant():
    user_id = request.args.get("user_id")
    summary = get_public_user_summary(user_id)
    if not summary:
        return jsonify({"error": "not found"}), 404
    system_prompt = f"You are assisting user {summary.get('display_name')}"
    raw_response = llm_chat(system_prompt)
    safe_response = sanitize_llm_output(raw_response)
    return jsonify({"response": safe_response})

Remediation pattern 3: Tighten Firestore rules and Flask access control

Ensure Firestore security rules prevent unauthorized reads of sensitive collections. In Flask, require authentication for endpoints that access Firestore and avoid exposing raw document IDs in URLs.

# Firestore rules (conceptual)
rules_version = '2';
service cloud.firestore {
  match /databases/{database}/documents {
    match /users/{userId} {
      allow read: if request.auth != null && request.auth.uid == userId;
    }
  }
}

Combine this with Flask-level checks and avoid sequential or guessable identifiers. middleBrick’s scans can validate these controls by checking unauthenticated access and BOLA/IDOR findings alongside LLM/AI Security tests.