Unicode Normalization in Django with Firestore

Unicode normalization inconsistencies between Django and Google Cloud Firestore can create security-relevant differences in how identifiers and user-controlled strings are compared and stored. Firestore stores strings as UTF-8 but does not enforce a canonical normalization form. Django, when handling user input, may apply normalization in certain contexts (e.g., form cleaning or model preprocessing) but does not do so uniformly across all data flows. This mismatch can lead to authentication bypass, IDOR, or privilege escalation when a normalized identifier (such as an email or username) is compared to a non-normalized stored value.

For example, consider an API endpoint that looks up a user document by email. An attacker can supply a carefully crafted email that is canonically equivalent to an authorized user’s email but byte-for-byte different. If Django normalizes the input before the lookup but Firestore stores the original, case-preserving value, the query may fail to match, or may match an unintended record depending on indexing and query behavior. This can enable BOLA/IDOR where one user gains access to another’s data through an alternate normalization form.

Another scenario involves filenames or document IDs that include Unicode combining characters. An attacker might upload or reference a file using a decomposed form (e.g., a\u0301 — a with combining acute) while the application expects precomposed forms (e.g., á). If access control checks operate on the raw identifier without normalization, an attacker can bypass path-based restrictions or enumeration protections. Insecure direct object references become more likely when IDs are derived from user-supplied strings that differ only in normalization but appear equivalent to humans.

Firestore’s indexing and query semantics compound the issue. Range queries and inequality filters on strings are affected by lexicographic ordering, which varies across normalization forms. A query that filters documents using a normalized string may not match entries stored in a different normalization form, leading to inconsistent visibility of data and potential information disclosure. Similarly, case sensitivity and accent sensitivity depend on how the client library serializes and sends strings, and Firestore does not normalize these automatically.

Because middleBrick tests unauthenticated attack surfaces and includes input validation checks, it can surface normalization-related inconsistencies by observing whether equivalent inputs produce different runtime behavior or data exposure. Findings will highlight discrepancies between submitted variants and stored representations, providing remediation guidance to enforce normalization consistently in Django before any Firestore interaction.

To mitigate Unicode normalization issues, normalize all user-controlled strings to a canonical form in Django before using them to construct Firestore queries or document IDs. Use Python’s unicodedata module to apply NFC or NFD consistently across your application. Choose one form (NFC is typical for web applications) and apply it at the boundary where data enters your system — for example, in form cleaning, model save methods, or API request preprocessing.

Code example: Normalizing email and document IDs before Firestore operations

import unicodedata
from google.cloud import firestore
from django.core.exceptions import ValidationError
from django.utils.encoding import force_str

def normalize_unicode(value: str) -> str:
    """Normalize Unicode input to NFC to ensure consistent comparison."""
    return unicodedata.normalize('NFC', force_str(value))

def get_user_by_email(email: str):
    """Retrieve a user document using a normalized email address."""
    normalized_email = normalize_unicode(email)
    db = firestore.Client()
    users_ref = db.collection('users')
    # Query using the normalized email; ensure Firestore index supports this query
    query = users_ref.where('email', '==', normalized_email).limit(1)
    results = query.stream()
    for doc in results:
        return doc.to_dict()
    return None

def safe_user_lookup(user_id: str):
    """Fetch a user by ID after normalizing to avoid IDOR via Unicode variants."""
    safe_id = normalize_unicode(user_id)
    db = firestore.Client()
    doc_ref = db.collection('users').document(safe_id)
    doc = doc_ref.get()
    if doc.exists:
        return doc.to_dict()
    return None

Code example: Normalizing data on model save

from django.db import models
import unicodedata

class UserProfile(models.Model):
    email = models.EmailField(unique=True)
    display_name = models.CharField(max_length=255)

    def clean(self):
        """Normalize strings before validation and storage."""
        self.email = unicodedata.normalize('NFC', self.email or '')
        self.display_name = unicodedata.normalize('NFC', self.display_name or '')
        super().clean()

    def save(self, *args, **kwargs):
        self.clean()
        super().save(*args, **kwargs)

Code example: Using a Firestore preprocessor utility

def prepare_firestore_document(data: dict) -> dict:
    """Recursively normalize string values in a dictionary destined for Firestore."""
    def _normalize(value):
        if isinstance(value, str):
            return unicodedata.normalize('NFC', value)
        elif isinstance(value, dict):
            return {k: _normalize(v) for k, v in value.items()}
        elif isinstance(value, list):
            return [_normalize(v) for v in value]
        return value
    return {k: _normalize(v) for k, v in data.items()}

# Usage before writing or querying
data = {'email': 'usér@example.com', 'tags': ['café', 'naïve']}
safe_data = prepare_firestore_document(data)

In addition to normalization, validate and constrain input to reduce unexpected equivalence classes. Use Django’s validators to restrict characters where appropriate and avoid relying on automatic escaping or encoding behavior. When integrating with Firestore, ensure that any indexes used in queries align with the chosen normalization form; composite and single-field indexes should reflect the canonical representation you enforce.

middleBrick can be used in the Pro plan to continuously monitor API endpoints for normalization-related inconsistencies across authenticated and unauthenticated surfaces. With continuous monitoring and configurable scanning schedules, you can detect regressions when new endpoints or fields introduce inconsistent handling. Findings include severity, remediation guidance, and mappings to relevant compliance frameworks such as OWASP API Top 10 and data protection regulations.