Log File Redaction

Application logs are essential for debugging, monitoring, and security analysis—but they inevitably capture personal information. User emails appear in authentication logs, customer names in transaction records, IP addresses in access logs, and sensitive data in error messages that expose request payloads. This creates a tension between operational needs and privacy requirements: developers need detailed logs to troubleshoot issues, while regulations like GDPR require minimization and protection of personal data.

Log redaction resolves this tension by automatically detecting and protecting PII while preserving the technical information needed for operations. Whether processing real-time log streams or remediating historical archives, intelligent redaction ensures your logs remain useful for their intended purpose without exposing sensitive data to unauthorized access or creating compliance violations.

PII in Logs

Common sources of personal information in logs:

Authentication Logs:

// Login events capture user identifiers
2024-01-15 10:30:00 INFO [auth] Login attempt: [email protected] ip=192.168.1.100
2024-01-15 10:30:01 ERROR [auth] Failed login for [email protected]: invalid password
2024-01-15 10:30:02 INFO [auth] Password reset requested for [email protected]

// PII present: email addresses, IP addresses

Application Logs:

// Request/response logging
2024-01-15 10:31:00 DEBUG [api] POST /users {"name":"John Smith","email":"[email protected]","ssn":"123-45-6789"}
2024-01-15 10:31:01 INFO [order] Order created for customer_id=12345 (Jane Doe, [email protected])
2024-01-15 10:31:02 ERROR [payment] Payment failed for card ending 4242, customer: Bob Wilson

// PII present: names, emails, SSN, partial card numbers

Error Logs:

// Exceptions often expose sensitive data
2024-01-15 10:32:00 ERROR [db] Query failed: SELECT * FROM users WHERE email='[email protected]'
2024-01-15 10:32:01 FATAL [app] Unhandled exception processing user John Smith (555-123-4567)
Stack trace: ...
  Request body: {"credit_card":"4111111111111111","cvv":"123"}

// PII in error context: queries, stack traces, request payloads

Access Logs:

// Web server access logs
192.168.1.100 - [email protected] [15/Jan/2024:10:33:00 +0000] "GET /account/profile HTTP/1.1" 200 1234
192.168.1.101 - - [15/Jan/2024:10:33:01 +0000] "POST /api/[email protected] HTTP/1.1" 201 89

// PII present: IP addresses, usernames, emails in URLs

Real-Time Log Processing

Process log streams as they're generated:

Streaming Architecture:

// Log source → Redaction → Log destination
Application → Fluentd/Logstash → RedactionAPI → Elasticsearch
                                      ↓
                              (Redacted stream)

// Processing flow
1. Log event generated by application
2. Log shipper collects event
3. Event sent to redaction service
4. PII detected and redacted
5. Clean event forwarded to storage/analysis

Logstash Filter Integration:

# logstash.conf
filter {
  http {
    url => "https://api.redactionapi.net/v1/redact/stream"
    verb => "POST"
    headers => {
      "Authorization" => "Bearer ${REDACTION_API_KEY}"
    }
    body => {
      "text" => "%{message}"
      "format" => "log"
    }
    target_body => "redacted"
  }

  mutate {
    replace => { "message" => "%{[redacted][text]}" }
  }
}

Fluent Bit Configuration:

[FILTER]
    Name         lua
    Match        *
    script       redact.lua
    call         redact_pii

[OUTPUT]
    Name         http
    Match        *
    Host         api.redactionapi.net
    Port         443
    URI          /v1/redact/batch
    Format       json
    tls          On

Log Format Support

Handle various log formats intelligently:

JSON Logs:

// Input
{"timestamp":"2024-01-15T10:30:00Z","level":"INFO","user":"[email protected]","action":"login","ip":"192.168.1.100"}

// Output (field-aware redaction)
{"timestamp":"2024-01-15T10:30:00Z","level":"INFO","user":"[EMAIL]","action":"login","ip":"[IP_ADDRESS]"}

// Structure preserved, PII redacted by field type

Syslog Format:

// RFC 5424 syslog
<165>1 2024-01-15T10:30:00Z myhost myapp 1234 - - User [email protected] logged in from 192.168.1.100

// Redacted
<165>1 2024-01-15T10:30:00Z myhost myapp 1234 - - User [EMAIL] logged in from [IP_ADDRESS]

// Syslog structure (priority, timestamp, host, app) preserved

Apache/Nginx Access Logs:

// Combined log format
192.168.1.100 - [email protected] [15/Jan/2024:10:30:00 +0000] "GET /user/profile HTTP/1.1" 200 1234 "https://example.com" "Mozilla/5.0..."

// Redacted
[IP_ADDRESS] - [EMAIL] [15/Jan/2024:10:30:00 +0000] "GET /user/profile HTTP/1.1" 200 1234 "https://example.com" "Mozilla/5.0..."

// Log analysis tools still parse correctly

Custom Formats:

// Define custom format patterns
{
  "format": "custom",
  "pattern": "{timestamp} [{level}] {message}",
  "fields": {
    "timestamp": {"type": "timestamp", "redact": false},
    "level": {"type": "loglevel", "redact": false},
    "message": {"type": "text", "redact": true}
  }
}

Preserving Log Utility

Maintain debugging and analysis capabilities:

Correlation ID Preservation:

// Correlation IDs link related log entries
{"trace_id":"abc123","user":"[email protected]","action":"checkout"}
{"trace_id":"abc123","user":"[email protected]","action":"payment"}
{"trace_id":"abc123","user":"[email protected]","action":"confirmation"}

// Redacted with consistent tokenization
{"trace_id":"abc123","user":"tok_user_001","action":"checkout"}
{"trace_id":"abc123","user":"tok_user_001","action":"payment"}
{"trace_id":"abc123","user":"tok_user_001","action":"confirmation"}

// Same user = same token for journey analysis

IP Address Handling Options:

// Full redaction
192.168.1.100 → [IP_ADDRESS]

// Partial masking (preserve network)
192.168.1.100 → 192.168.1.xxx

// Geographic preservation
192.168.1.100 → [IP_ADDRESS:US:CA:San Francisco]

// Hashing (consistent reference)
192.168.1.100 → ip_hash_a1b2c3d4

Timestamp Precision:

// Preserve exact timestamps for debugging
// Optional: Generalize for additional privacy

"2024-01-15T10:30:45.123Z" → "2024-01-15T10:30:45.123Z" // Keep
"2024-01-15T10:30:45.123Z" → "2024-01-15T10:30:00Z"     // Minute precision
"2024-01-15T10:30:45.123Z" → "2024-01-15"               // Date only

Context-Aware Processing

Avoid false positives in technical contexts:

// Technical data that looks like PII but isn't

// UUIDs (not personal identifiers)
user_id: 550e8400-e29b-41d4-a716-446655440000  // Don't redact

// Numeric codes in technical contexts
error_code: 123-45-6789  // Not an SSN in this context

// Hostnames with email-like patterns
smtp.example.com  // Not an email address

// Version numbers
version: 1.234.5  // Not a phone number

// Context indicators help avoid false positives:
- Field names (error_code vs ssn)
- Surrounding text ("version" vs "phone")
- Format validation (UUID regex vs SSN regex)

Log Management Integration

Elasticsearch/ELK Stack:

// Logstash pipeline with redaction
input {
  beats { port => 5044 }
}

filter {
  # Parse log format
  grok {
    match => { "message" => "%{COMBINEDAPACHELOG}" }
  }

  # Redact PII fields
  http {
    url => "https://api.redactionapi.net/v1/redact"
    body => {
      "fields" => {
        "clientip" => "%{clientip}",
        "auth" => "%{auth}",
        "request" => "%{request}"
      }
    }
  }
}

output {
  elasticsearch { hosts => ["elasticsearch:9200"] }
}

Splunk Integration:

# Splunk HEC with preprocessing
# Use Splunk's modular inputs or HEC event modification

# props.conf - transform before indexing
[source::myapp_logs]
TRANSFORMS-redact = redact_pii

# transforms.conf
[redact_pii]
REGEX = email=([^\s]+)
FORMAT = email=[EMAIL]
DEST_KEY = _raw

Datadog Integration:

// Datadog Log Pipeline with redaction
// Use Datadog's Sensitive Data Scanner or preprocess

// Agent configuration
logs:
  - type: file
    path: /var/log/myapp/*.log
    processing_rules:
      - type: mask_sequences
        name: redact_emails
        pattern: \b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b
        replace_placeholder: "[EMAIL]"

Historical Log Processing

Remediate existing log archives:

// Batch process historical logs
const archiveJob = await redactionClient.createBatchJob({
  source: {
    type: 's3',
    bucket: 'company-log-archives',
    prefix: 'logs/2023/',
    pattern: '*.log.gz'
  },
  processing: {
    format: 'auto_detect',
    compression: 'gzip',
    parallelism: 10
  },
  output: {
    bucket: 'company-log-archives-redacted',
    prefix: 'logs/2023/',
    preserveStructure: true
  }
});

// Monitor progress
while (job.status !== 'completed') {
  const status = await redactionClient.getJobStatus(archiveJob.id);
  console.log(`Processed: ${status.filesProcessed}/${status.totalFiles}`);
  await sleep(10000);
}

Compliance Considerations

GDPR Log Requirements:

• Minimize personal data in logs (data minimization principle)
• Define and enforce retention periods
• Support data subject access and deletion requests
• Protect logs as personal data if they contain PII

Security Logging Requirements:

• Many frameworks require security event logging
• Balance between security monitoring and privacy
• Redaction can enable logging without privacy violation
• Tokenization preserves correlation for security analysis

API Usage

// Stream processing endpoint
POST /v1/redact/logs
Content-Type: application/x-ndjson

{"timestamp":"2024-01-15T10:30:00Z","message":"User [email protected] logged in"}
{"timestamp":"2024-01-15T10:30:01Z","message":"Order placed by Jane Doe (555-1234)"}

Response (streaming):
{"timestamp":"2024-01-15T10:30:00Z","message":"User [EMAIL] logged in"}
{"timestamp":"2024-01-15T10:30:01Z","message":"Order placed by [NAME] ([PHONE])"}

// Batch processing for files
POST /v1/redact/file
{
  "file_url": "s3://logs/application.log.gz",
  "format": "auto",
  "compression": "gzip"
}