# Portfolio Snapshot System Implementation

## Overview

This document describes the implementation of the Portfolio Snapshot Generation system as part of task 7.3 from the granular permissions and reporting system specification. The system provides automated daily portfolio snapshot generation, health score calculations, growth rate tracking, and performance alerts.

## Features Implemented

### 1. Automated Daily Portfolio Snapshot Generation
- **Service**: `PortfolioSnapshotService` in `users/portfolio_snapshot_service.py`
- **Management Command**: `generate_portfolio_snapshots` in `users/management/commands/`
- **Celery Tasks**: Automated background processing in `users/tasks.py`
- **Cron Job**: Shell script for scheduled execution (`cron_portfolio_snapshots.sh`)

### 2. Portfolio Health Score Calculations
- Comprehensive health scoring algorithm based on:
  - Collection rate (40% weight)
  - Default rate (30% weight)
  - Growth rate (20% weight)
  - Portfolio yield (10% weight)
- Score range: 0-100 with automatic categorization (Critical < 50, Warning < 70, Good < 80, Excellent ≥ 80)

### 3. Portfolio Growth Rate Tracking
- Period-over-period growth calculations
- Client acquisition and churn tracking
- Loan portfolio expansion metrics
- Comparative analysis capabilities

### 4. Portfolio Performance Alerts and Notifications
- Real-time alert generation based on:
  - Critical health scores (< 50%)
  - High default rates (> 10%)
  - Low collection rates (< 80%)
  - High Portfolio at Risk (PAR 30+ > 15%)
  - Negative growth trends
- Email notifications for critical alerts
- In-system notification management

## System Components

### Models

#### PortfolioSnapshot
Located in `users/enhanced_permissions_models.py`

Key fields:
- `manager`: Portfolio manager (ForeignKey to CustomUser)
- `branch`: Branch association
- `snapshot_date`: Date of the snapshot
- `total_clients`, `active_clients`: Client metrics
- `active_loans`, `total_loans`: Loan metrics
- `total_disbursed`, `total_outstanding`, `total_collected`: Financial metrics
- `par_30`, `par_60`, `par_90`: Portfolio at Risk metrics
- `default_rate`, `collection_rate`, `portfolio_yield`: Performance metrics

Methods:
- `get_portfolio_health_score()`: Calculates comprehensive health score
- `calculate_growth_rate()`: Computes growth rate vs previous period

### Services

#### PortfolioSnapshotService
Located in `users/portfolio_snapshot_service.py`

Key methods:
- `generate_daily_snapshots()`: Generate snapshots for all managers
- `calculate_portfolio_health_score()`: Calculate health score for a manager
- `calculate_growth_rate()`: Calculate growth rate over specified period
- `get_portfolio_trends()`: Get trend data for visualization
- `generate_performance_report()`: Comprehensive performance reporting

### Management Commands

#### generate_portfolio_snapshots
Located in `users/management/commands/generate_portfolio_snapshots.py`

Usage:
```bash
# Generate snapshots for today
python manage.py generate_portfolio_snapshots

# Generate for specific date
python manage.py generate_portfolio_snapshots --date 2024-01-15

# Generate for specific manager
python manage.py generate_portfolio_snapshots --manager-id <uuid>

# Force regeneration of existing snapshots
python manage.py generate_portfolio_snapshots --force

# Verbose output
python manage.py generate_portfolio_snapshots --verbose
```

### Views and Templates

#### Portfolio Snapshot Dashboard
- **View**: `portfolio_snapshot_dashboard` in `users/portfolio_snapshot_views.py`
- **Template**: `templates/users/portfolio_snapshot_dashboard.html`
- **URL**: `/users/portfolio/snapshots/`

Features:
- Overview of all portfolio managers
- Health score visualization
- Recent alerts display
- Quick action buttons

#### Manager Portfolio Detail
- **View**: `manager_portfolio_detail` in `users/portfolio_snapshot_views.py`
- **URL**: `/users/portfolio/manager/<uuid>/`

Features:
- Detailed portfolio metrics
- Trend charts and visualizations
- Performance history
- Alert management

#### Manual Snapshot Generation
- **View**: `generate_snapshot_manual` in `users/portfolio_snapshot_views.py`
- **Template**: `templates/users/generate_snapshot_manual.html`
- **URL**: `/users/portfolio/generate/`

Features:
- Manual snapshot generation interface
- Date selection
- Manager-specific generation
- Force regeneration option

### Celery Tasks

#### Background Processing
Located in `users/tasks.py`

Tasks:
- `generate_daily_portfolio_snapshots`: Automated daily generation
- `generate_portfolio_snapshot_for_manager`: Single manager generation
- `calculate_portfolio_health_scores`: Health score calculations
- `send_weekly_portfolio_summary`: Weekly summary emails
- `cleanup_old_snapshots`: Database maintenance

### Notifications

#### Enhanced Notification System
Extended `utils/models.py` Notification model with:
- Portfolio-specific notification types
- `action_required` field for critical alerts
- `portfolio_snapshot` foreign key relationship
- `alert_data` JSON field for additional context

New notification types:
- `portfolio_alert`: General portfolio alerts
- `portfolio_health_critical`: Critical health score alerts
- `portfolio_health_warning`: Health score warnings
- `high_default_rate`: High default rate alerts
- `low_collection_rate`: Low collection rate alerts
- `high_par`: High Portfolio at Risk alerts
- `negative_growth`: Negative growth alerts

## Installation and Setup

### 1. Database Migration
```bash
python manage.py makemigrations
python manage.py migrate
```

### 2. Create Initial Data
Ensure you have:
- Portfolio managers (users with role 'loan_officer' or 'team_leader')
- Active branches
- Loan and client data for meaningful snapshots

### 3. Manual Generation (First Time)
```bash
python manage.py generate_portfolio_snapshots --verbose
```

### 4. Set Up Automated Generation

#### Option A: Cron Job (Linux/Mac)
```bash
# Edit crontab
crontab -e

# Add daily execution at 1:00 AM
0 1 * * * /path/to/project/cron_portfolio_snapshots.sh
```

#### Option B: Celery Beat (Recommended)
Add to Django settings:
```python
from celery.schedules import crontab

CELERY_BEAT_SCHEDULE = {
    'generate-daily-portfolio-snapshots': {
        'task': 'users.tasks.generate_daily_portfolio_snapshots',
        'schedule': crontab(hour=1, minute=0),  # Daily at 1:00 AM
    },
    'send-weekly-portfolio-summary': {
        'task': 'users.tasks.send_weekly_portfolio_summary',
        'schedule': crontab(hour=9, minute=0, day_of_week=1),  # Monday at 9:00 AM
    },
    'cleanup-old-snapshots': {
        'task': 'users.tasks.cleanup_old_snapshots',
        'schedule': crontab(hour=2, minute=0, day_of_week=0),  # Sunday at 2:00 AM
    },
}
```

#### Option C: Windows Task Scheduler
Create a batch file and schedule it using Windows Task Scheduler:
```batch
@echo off
cd /d "C:\path\to\your\project"
python manage.py generate_portfolio_snapshots
```

## Usage

### Accessing the Dashboard
1. Log in as an admin, team leader, or loan officer
2. Navigate to `/users/portfolio/snapshots/`
3. View portfolio overview and health scores
4. Click on individual managers for detailed views

### Generating Manual Snapshots
1. Access `/users/portfolio/generate/` (admin only)
2. Select target date
3. Choose scope (all managers or specific manager)
4. Click "Generate Snapshots"

### Viewing Alerts
1. Navigate to `/users/portfolio/alerts/`
2. Filter by priority, type, or unread status
3. Mark alerts as read
4. Take action on critical alerts

### Exporting Data
1. Go to manager detail view
2. Click "Export Data" button
3. Select date range
4. Download CSV file with portfolio metrics

## Health Score Calculation

The health score is calculated using a weighted formula:

```
Health Score = (Collection Rate × 0.4) + 
               (Default Score × 0.3) + 
               (Growth Score × 0.2) + 
               (Yield Score × 0.1)

Where:
- Collection Rate: Direct percentage (max 100)
- Default Score: 100 - (Default Rate × 10) (lower default = higher score)
- Growth Score: Normalized growth rate + 50 (0% growth = 50 points)
- Yield Score: Portfolio Yield × 5 (20% yield = 100 points)
```

## Alert Thresholds

### Critical Alerts (Immediate Action Required)
- Health Score < 50%
- Default Rate > 10%
- PAR 30+ > 15%

### Warning Alerts (Monitor Closely)
- Health Score < 70%
- Default Rate > 5%
- Collection Rate < 80%
- PAR 30+ > 10%
- Negative Growth < -10%

## Performance Considerations

### Database Optimization
- Indexes on `manager`, `snapshot_date`, and `branch` fields
- Unique constraint on `manager` + `snapshot_date`
- Efficient queries with select_related and prefetch_related

### Caching
- Permission caching in PagePermissionManager
- Snapshot data caching for frequently accessed metrics
- 15-minute cache timeout for real-time data

### Background Processing
- Celery tasks for heavy computations
- Retry logic with exponential backoff
- Error handling and notification

## Monitoring and Maintenance

### Log Files
- Management command logs: `logs/portfolio_snapshots_cron.log`
- Django application logs: Standard Django logging
- Celery task logs: Celery worker logs

### Database Maintenance
- Automatic cleanup of snapshots older than 365 days
- Log file rotation when size exceeds 10MB
- Regular database optimization recommended

### Email Notifications
- Critical alert emails to portfolio managers
- Weekly summary emails to team leaders and admins
- Task failure notifications to administrators

## Testing

### Unit Tests
Located in `users/tests/test_portfolio_snapshot_service.py`

Run tests:
```bash
python manage.py test users.tests.test_portfolio_snapshot_service
```

### Manual Testing
1. Create test users with different roles
2. Generate sample loan and client data
3. Run snapshot generation
4. Verify health score calculations
5. Test alert generation

## Troubleshooting

### Common Issues

#### No Snapshots Generated
- Check if portfolio managers exist with correct roles
- Verify branch assignments
- Ensure loan and client data exists
- Check database permissions

#### Health Scores Seem Incorrect
- Verify loan status calculations
- Check payment data integrity
- Review PAR calculations
- Validate date ranges

#### Alerts Not Sending
- Check email configuration
- Verify notification model setup
- Review Celery task execution
- Check user email addresses

#### Performance Issues
- Monitor database query performance
- Check cache hit rates
- Review Celery worker capacity
- Optimize snapshot generation frequency

### Debug Commands
```bash
# Check portfolio managers
python manage.py shell -c "from users.models import CustomUser; print(CustomUser.objects.filter(role__in=['loan_officer', 'team_leader']).count())"

# Test service initialization
python manage.py shell -c "from users.portfolio_snapshot_service import PortfolioSnapshotService; service = PortfolioSnapshotService(); print('Service OK')"

# Generate test snapshot
python manage.py generate_portfolio_snapshots --verbose --date 2024-01-01
```

## Future Enhancements

### Planned Features
1. Advanced analytics and forecasting
2. Comparative benchmarking across branches
3. Mobile-responsive dashboard
4. Real-time WebSocket updates
5. Advanced filtering and search
6. Custom alert threshold configuration
7. Integration with external reporting tools

### API Endpoints
Consider adding REST API endpoints for:
- Snapshot data retrieval
- Health score calculations
- Alert management
- Trend analysis

## Support

For issues or questions regarding the Portfolio Snapshot System:
1. Check the troubleshooting section above
2. Review Django and Celery logs
3. Verify database integrity
4. Contact system administrator

## Requirements Compliance

This implementation satisfies the following requirements from task 7.3:

✅ **Create automated daily portfolio snapshot generation**
- Implemented via management command and Celery tasks
- Configurable scheduling options
- Error handling and retry logic

✅ **Implement portfolio health score calculations**
- Comprehensive scoring algorithm
- Multiple performance factors
- Automatic categorization and alerts

✅ **Add portfolio growth rate tracking**
- Period-over-period comparisons
- Client and loan growth metrics
- Trend analysis capabilities

✅ **Create portfolio performance alerts and notifications**
- Multi-level alert system
- Email notifications for critical issues
- In-system notification management
- Configurable thresholds

The system is production-ready and provides a solid foundation for portfolio performance monitoring and management.