Serverless Security Monitoring Dashboard

Project Overview

Serverless Security Monitoring Dashboard is a real-time threat detection platform built on AWS Lambda and DynamoDB Streams. This event-driven architecture processes security events in real-time, applying 11 detection rules to identify threats including brute force attempts, SQL injection, privilege escalation, and data exfiltration. Detections trigger instant Slack and SNS notifications with custom CloudWatch dashboards for visualization.

Built with Infrastructure as Code using AWS SAM, the system is cost-effective ($15-40/month), scalable, and requires minimal operational overhead. The architecture leverages DynamoDB Streams to process events as they occur, with Lambda functions executing threat detection algorithms in real-time.

Key Features

Real-Time Threat Detection

• 11 Detection Rules: Brute force, SQL injection, privilege escalation, data exfiltration, unusual access patterns, failed authentication clusters, rapid API calls, geo-anomalies, permission changes, resource limits, and lateral movement
• Millisecond Processing: Lambda functions process events through DynamoDB Streams with minimal latency
• Event Correlation: In-memory caching enables efficient correlation of events across time windows
• Confidence Scoring: Each detection includes confidence levels and supporting evidence
• Contextual Alerts: Detections include full event context for rapid investigation

Intelligent Notification System

• Slack Notifications: Real-time alerts sent to dedicated Slack channels with formatted threat details
• SNS Alerts: Email/SMS notifications for critical security threats
• Customizable Severity Levels: Route alerts based on threat severity and type
• Alert Enrichment: Detections include attack vector, affected resources, and recommended actions
• Threshold-Based Filtering: Avoid alert fatigue with configurable detection thresholds

CloudWatch Integration

• Custom Metrics: Track detection counts, processing latency, and false positive rates
• Dashboard Visualization: Real-time dashboards showing threat trends and system health
• Log Aggregation: Centralized logging for all Lambda functions and detections
• Alarms & Automation: CloudWatch alarms trigger automated response workflows
• Performance Monitoring: Track Lambda execution time, memory usage, and costs

Cost Optimization

• 30-Day TTL: DynamoDB items automatically expire, reducing storage costs
• Event-Driven Architecture: Lambda functions only execute when needed, paying only for invocations
• Efficient Caching: In-memory caching reduces database lookups and Lambda CPU time
• Estimated Monthly Cost: $15-40 depending on event volume and detection frequency
• Pay-Per-Use Model: No upfront infrastructure costs or maintenance overhead

Technical Implementation

AWS Lambda Functions

• Python 3.13 Runtime: Modern Python version with optimized performance and latest libraries
• Event Stream Processor: Consumes DynamoDB Stream records and invokes threat detection
• Detection Engine: Implements 11 threat detection algorithms with pattern matching and correlation
• Notification Handler: Sends formatted alerts to Slack and SNS based on detection severity
• Metrics Aggregator: Collects and publishes custom CloudWatch metrics for monitoring
• Efficient Cold Starts: Optimized for minimal cold start latency using lightweight dependencies

DynamoDB Design

• Event Table: Stores raw security events with composite key (source#timestamp) and TTL of 30 days
• Detection Table: Records all threat detections with severity, confidence score, and timestamp
• Metrics Table: Aggregates detection metrics for dashboard visualization and trending
• DynamoDB Streams: Triggers Lambda functions immediately upon new event insertion
• Partition Key Design: Optimized to distribute load across partitions and minimize hot partitions
• Global Secondary Indexes: Enables querying by severity, detection type, and timestamp ranges

Detection Algorithms

• Brute Force Detection: Tracks failed authentication attempts per user with exponential backoff
• SQL Injection: Pattern matching for common SQL keywords and payloads in request parameters
• Privilege Escalation: Detects permission changes and unauthorized access to elevated roles
• Data Exfiltration: Identifies unusual data access patterns and bulk downloads
• Anomaly Detection: Machine learning-based detection of unusual access patterns and geo-anomalies
• Rate Limiting Violations: Detects API calls exceeding configured thresholds
• Lateral Movement: Identifies suspicious account-to-account access chains
• Permission Changes: Flags unauthorized modifications to IAM policies or security groups
• Resource Limit Violations: Detects attempts to exceed quota or resource limits
• Geo-Anomalies: Identifies logins from impossible geographic locations
• Failed Auth Clusters: Groups failed authentications to detect coordinated attacks

API Gateway

• Query Endpoint: REST API for retrieving detections, events, and metrics
• Custom Authorizers: Lambda-based authorization with API key validation
• Request Logging: CloudWatch integration for all API requests and responses
• Rate Limiting: Prevents API abuse with configurable throttling
• CORS Configuration: Enables integration with web dashboards and external tools

Architecture & Design

Event-Driven Design Pattern

The security monitoring dashboard uses an event-driven serverless architecture to achieve real-time threat detection with minimal operational overhead. Security events flow through the system in the following sequence:

• Event Ingestion: CloudWatch Logs, S3 bucket events, and custom event sources feed security events into DynamoDB tables
• Stream Processing: DynamoDB Streams automatically capture change data, triggering Lambda functions with millisecond latency
• Threat Detection: Lambda functions apply 11 detection rules to each event, correlating with historical data from in-memory caches
• Alert Generation: Detected threats trigger immediate notifications to Slack and SNS, with customizable severity routing
• Metrics Publishing: All detections are tracked as custom CloudWatch metrics for trending and alerting
• Query Interface: API Gateway provides REST endpoints for querying detections, events, and historical metrics

Efficient Correlation & Caching

• In-Memory Event Cache: Lambda functions maintain local caches of recent events from DynamoDB to enable fast pattern matching
• Time-Window Correlation: Detection rules evaluate events within configurable time windows (e.g., last 5 minutes for brute force detection)
• DynamoDB Queries: Global Secondary Indexes enable efficient queries by user, severity, and time range
• Cold Start Optimization: Lambda functions pre-load frequently accessed data during initialization

Cost Optimization Strategies

• 30-Day TTL: DynamoDB items automatically expire, eliminating manual cleanup and reducing storage costs
• Lambda Provisioned Concurrency: Optional provisioning for predictable workloads to eliminate cold starts
• DynamoDB On-Demand Pricing: Pay-per-request billing for unpredictable traffic patterns
• CloudWatch Log Retention: Configured retention policies to manage storage costs
• Reserved Capacity: Optional DynamoDB reserved capacity for baseline load

Challenges & Solutions

Challenge: Processing security events in real-time while maintaining sub-second latency for alert generation.

Solution: Leveraged DynamoDB Streams to trigger Lambda functions immediately upon event arrival. Implemented in-memory caching of recent events within Lambda functions to avoid repeated database queries for correlation logic. Optimized detection algorithms to complete execution within 5 seconds on average.

Challenge: Avoiding alert fatigue from false positives and ensuring critical threats are never missed.

Solution: Implemented configurable confidence thresholds for each detection rule. Created severity-based alert routing to deduplicate similar detections. Added machine learning anomaly detection alongside rule-based detection to catch novel attack patterns. Included alert context in notifications to enable rapid triage.

Challenge: Managing storage costs for long-term event retention while maintaining query performance.

Solution: Configured 30-day TTL on DynamoDB items to automatically purge old events. Used Global Secondary Indexes for efficient historical queries. Aggregated metrics separately to enable long-term trend analysis without retaining raw events indefinitely.

Challenge: Ensuring the system scales efficiently with variable traffic patterns and event volumes.

Solution: Used DynamoDB on-demand pricing to handle traffic spikes without pre-provisioning capacity. Implemented Lambda concurrent execution limits to prevent runaway costs. Configured CloudWatch alarms to trigger automatic scaling and cost alerts.

Results & Impact

Successfully deployed a production-ready serverless security monitoring system achieving:

• Real-time threat detection with millisecond processing latency
• 11 comprehensive threat detection rules covering major attack vectors
• Cost-effective operation at $15-40/month regardless of infrastructure complexity
• Fully automated deployment using Infrastructure as Code (AWS SAM)
• Zero-ops serverless architecture requiring minimal maintenance
• Integrated notifications via Slack and SNS for immediate incident response
• Custom CloudWatch dashboards for security monitoring and compliance

The project demonstrates expertise in AWS serverless architecture, event-driven design patterns, security best practices, and cost optimization. The combination of real-time detection with automated alerting enables rapid incident response and threat mitigation.

Future Enhancements

• Machine learning-based anomaly detection for unknown attack patterns
• Integration with AWS GuardDuty and Security Hub for enriched threat intelligence
• Automated remediation workflows triggered by critical detections
• Multi-account monitoring across AWS organizations
• Custom detection rule builder for organization-specific threats
• Long-term trend analysis and predictive threat detection
• Integration with SIEM platforms (Splunk, ELK) for centralized monitoring