PII Detection

Semantic Router provides built-in Personally Identifiable Information (PII) detection to protect sensitive data in user queries. The system uses fine-tuned BERT models to identify and handle various types of PII according to configurable policies.

Overview

The PII detection system:

Identifies common PII types in user queries
Enforces model-specific PII policies
Blocks or masks sensitive information based on configuration
Filters model candidates based on PII compliance
Logs policy violations for monitoring

Supported PII Types

The system can detect the following PII types:

PII Type	Description	Examples
`PERSON`	Person names	"John Smith", "Mary Johnson"
`EMAIL_ADDRESS`	Email addresses	"user@example.com"
`PHONE_NUMBER`	Phone numbers	"+1-555-123-4567", "(555) 123-4567"
`US_SSN`	US Social Security Numbers	"123-45-6789"
`STREET_ADDRESS`	Physical addresses	"123 Main St, New York, NY"
`GPE`	Geopolitical entities	Countries, states, cities
`ORGANIZATION`	Organization names	"Microsoft", "OpenAI"
`CREDIT_CARD`	Credit card numbers	"4111-1111-1111-1111"
`US_DRIVER_LICENSE`	US Driver's License	"D123456789"
`IBAN_CODE`	International Bank Account Number	"GB82 WEST 1234 5698 7654 32"
`IP_ADDRESS`	IP addresses	"192.168.1.1", "2001:db8::1"
`DOMAIN_NAME`	Domain/website names	"example.com", "google.com"
`DATE_TIME`	Date/time information	"2024-01-15", "January 15th"
`AGE`	Age information	"25 years old", "born in 1990"
`NRP`	Nationality/Religious/Political groups	"American", "Christian", "Democrat"
`ZIP_CODE`	ZIP/postal codes	"10001", "SW1A 1AA"

Configuration

Basic PII Detection

Enable PII detection in your configuration:

# config/config.yaml
classifier:
  pii_model:
    model_id: "models/pii_classifier_modernbert-base_model"
    threshold: 0.7                 # Global detection threshold (0.0-1.0)
    use_cpu: true                  # Run on CPU
    pii_mapping_path: "config/pii_type_mapping.json"  # Path to PII type mapping

Category-Level PII Detection

New in v0.x: Configure PII detection thresholds at the category level for fine-grained control based on category-specific requirements and consequences.

# Global PII configuration - applies to all categories by default
classifier:
  pii_model:
    model_id: "models/pii_classifier_modernbert-base_model"
    threshold: 0.7  # Global default threshold
    use_cpu: true
    pii_mapping_path: "config/pii_type_mapping.json"

# Category-specific PII settings
categories:
  # Healthcare category: High threshold for critical PII
  - name: healthcare
    description: "Healthcare and medical queries"
    pii_enabled: true       # Enable PII detection (default: inherits from global)
    pii_threshold: 0.9      # Higher threshold for stricter detection
    model_scores:
      - model: secure-llm
        score: 0.9
        use_reasoning: false

  # Finance category: Very high threshold for financial PII
  - name: finance
    description: "Financial queries"
    pii_enabled: true
    pii_threshold: 0.95     # Very strict for SSN, credit cards, etc.
    model_scores:
      - model: secure-llm
        score: 0.9
        use_reasoning: false

  # Code generation: Lower threshold to reduce false positives
  - name: code_generation
    description: "Code and technical content"
    pii_enabled: true
    pii_threshold: 0.5      # Lower to avoid flagging code artifacts as PII
    model_scores:
      - model: general-llm
        score: 0.9
        use_reasoning: true

  # Testing: Disable PII detection
  - name: testing
    description: "Test scenarios"
    pii_enabled: false      # Disable for testing
    model_scores:
      - model: general-llm
        score: 0.6
        use_reasoning: false

  # General: Uses global settings
  - name: general
    description: "General queries"
    # pii_enabled and pii_threshold not specified - inherits global settings
    model_scores:
      - model: general-llm
        score: 0.5
        use_reasoning: false

Configuration Inheritance:

pii_enabled: If not specified, inherits from global PII model configuration (enabled if pii_model is configured)
pii_threshold: If not specified, inherits from classifier.pii_model.threshold

Threshold Guidelines by Category:

Critical categories (healthcare, finance, legal): 0.9-0.95 - Strict detection, fewer false positives
Customer-facing (support, sales): 0.75-0.85 - Balanced detection
Internal tools (code, testing): 0.5-0.65 - Relaxed to reduce false positives
Public content (docs, marketing): 0.6-0.75 - Broader detection before publication

Model-Specific PII Policies

Configure different PII policies for different models:

# vLLM endpoints configuration
vllm_endpoints:
  - name: secure-model
    address: "127.0.0.1"
    port: 8080
  - name: general-model
    address: "127.0.0.1"
    port: 8081

# Model-specific configurations
model_config:
  secure-llm:
    pii_policy:
      allow_by_default: false      # Block all PII by default
      pii_types:                   # Only allow these specific types
        - "EMAIL_ADDRESS"
        - "GPE"
        - "ORGANIZATION"

  general-llm:
    pii_policy:
      allow_by_default: true       # Allow all PII by default
      pii_types: []                # Not used when allow_by_default is true

How PII Detection Works

The PII detection system works as follows:

Detection: The PII classifier model analyzes incoming text to identify PII types
Policy Check: The system checks if the detected PII types are allowed for the target model
Routing Decision: Models that don't allow the detected PII types are filtered out
Logging: All PII detections and policy decisions are logged for monitoring

API Integration

PII detection is automatically integrated into the routing process. When a request is made to the router, the system:

Analyzes the input text for PII using the configured classifier
Checks PII policies for candidate models
Filters out models that don't allow the detected PII types
Routes to an appropriate model that can handle the PII

Note: The current implementation uses the global PII threshold during automatic routing. To use category-specific thresholds, you can:

Configure thresholds appropriately for each category in your config
Access category-specific thresholds using config.GetPIIThresholdForCategory(categoryName) in your code
Call classifier.ClassifyPIIWithThreshold(text, threshold) with the category-specific threshold when you have category context

Classification Endpoint

You can also check PII detection directly using the classification API:

curl -X POST http://localhost:8080/api/v1/classify \
  -H "Content-Type: application/json" \
  -d '{
    "text": "My email is john.doe@example.com and I live in New York"
  }'

The response includes PII information along with category classification results.

Monitoring and Metrics

The system exposes PII-related metrics:

# Prometheus metrics
pii_detections_total{type="EMAIL_ADDRESS"} 45
pii_detections_total{type="PERSON"} 23
pii_policy_violations_total{model="secure-model"} 12
pii_requests_blocked_total 8
pii_requests_masked_total 15

Best Practices

1. Threshold Tuning

Start with threshold: 0.7 for balanced accuracy
Increase to 0.8-0.9 for high-security environments
Decrease to 0.5-0.6 for broader detection
Use category-level thresholds for fine-grained control based on PII type consequences

Category-Specific Threshold Guidelines

Different categories have different PII sensitivity requirements:

Critical Categories (Healthcare, Finance, Legal):

Threshold: 0.9-0.95
Rationale: High precision required; false positives on medical/financial terms are costly
Example PII: SSN, Credit Cards, Medical Records
Risk if too low: Too many false positives disrupt workflows

Customer-Facing Categories (Support, Sales):

Threshold: 0.75-0.85
Rationale: Balance between catching PII and avoiding false positives
Example PII: Email, Phone, Names, Addresses
Risk if too low: Moderate false positive rate

Internal Tools (Code Generation, Development):

Threshold: 0.5-0.65
Rationale: Code/technical content often triggers false positives; lower threshold needed
Example PII: Variable names, test data that looks like PII
Risk if too high: May still flag harmless code artifacts

Public Content (Documentation, Marketing):

Threshold: 0.6-0.75
Rationale: Broader detection before publication; acceptable to review more false positives
Example PII: Author names, example emails, placeholder data
Risk if too high: May miss PII that could be published

2. Policy Design

Use allow_by_default: false for sensitive models
Explicitly list allowed PII types for clarity
Consider different policies for different use cases
Combine category-level thresholds with model-level policies for defense in depth

3. Action Selection

Use block for high-security scenarios
Use mask when processing is still needed
Use allow with logging for audit requirements

4. Model Filtering

Configure PII policies to automatically filter model candidates
Ensure at least one model can handle each PII scenario
Test policy combinations thoroughly

Troubleshooting

Common Issues

High False Positives

Lower the detection threshold
Review training data for edge cases
Consider custom model fine-tuning

Missed PII Detection

Increase detection sensitivity
Check if PII type is supported
Verify model is properly loaded

Policy Conflicts

Ensure at least one model allows detected PII types
Check allow_by_default settings
Review pii_types_allowed lists

Debug Mode

Enable detailed PII logging:

logging:
  level: debug
  pii_detection: true

This will log all PII detection decisions and policy evaluations.

Overview​

Supported PII Types​

Configuration​

Basic PII Detection​

Category-Level PII Detection​

Model-Specific PII Policies​

How PII Detection Works​

API Integration​

Classification Endpoint​

Monitoring and Metrics​

Best Practices​

1. Threshold Tuning​

Category-Specific Threshold Guidelines​

2. Policy Design​

3. Action Selection​

4. Model Filtering​

Troubleshooting​

Common Issues​

Debug Mode​