# Performance Testing Guide
## Task 13.4: Performance Testing

**Purpose:** Verify that all features meet the performance requirements specified in the design document.

## Performance Requirements

From the requirements document:
1. Disbursed loans report SHALL load within 3 seconds for datasets up to 500 loans
2. Age & gender analytics SHALL calculate within 5 seconds for datasets up to 1000 applications
3. PDF export SHALL generate within 30 seconds for reports up to 1000 records
4. Excel export SHALL generate within 20 seconds for reports up to 1000 records

## Test Environment Setup

### Prerequisites
- [ ] Test database with sufficient data
- [ ] Development server running
- [ ] Performance monitoring tools installed
- [ ] Baseline measurements recorded

### Data Generation Script

```python
# generate_test_data.py
"""
Generate test data for performance testing.
"""
import os
import django
from decimal import Decimal
from datetime import datetime, timedelta
import random

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'branch_system.settings')
django.setup()

from loans.models import Loan, LoanApplication, LoanProduct
from users.models import CustomUser, Branch

def generate_test_loans(count=500):
    """Generate test loans for performance testing"""
    print(f"Generating {count} test loans...")
    
    # Get or create test data
    branch = Branch.objects.first()
    if not branch:
        branch = Branch.objects.create(name="Test Branch", code="TB001")
    
    borrower = CustomUser.objects.filter(role='borrower').first()
    if not borrower:
        borrower = CustomUser.objects.create_user(
            username='perf_test_borrower',
            email='perf@test.com',
            password='testpass123',
            role='borrower',
            branch=branch,
            phone_number='+254700000000'
        )
    
    product = LoanProduct.objects.first()
    if not product:
        product = LoanProduct.objects.create(
            name='Test Product',
            product_type='boost',
            min_amount=Decimal('1000'),
            max_amount=Decimal('100000'),
            interest_rate=Decimal('10.0'),
            processing_fee=Decimal('5.0'),
            min_duration=7,
            max_duration=90,
            available_repayment_methods=['monthly']
        )
    
    # Generate loans
    statuses = ['active', 'paid', 'defaulted', 'rolled_over', 'written_off']
    
    for i in range(count):
        # Random amounts
        principal = Decimal(random.randint(5000, 50000))
        interest = principal * Decimal('0.10')
        processing_fee = principal * Decimal('0.05')
        total = principal + interest + processing_fee
        
        # Random dates
        days_ago = random.randint(0, 365)
        disbursement_date = datetime.now() - timedelta(days=days_ago)
        due_date = disbursement_date + timedelta(days=30)
        
        # Create application
        app = LoanApplication.objects.create(
            application_number=f'PERF-APP-{i:06d}',
            borrower=borrower,
            loan_product=product,
            requested_amount=principal,
            requested_duration=30,
            purpose='Performance test',
            status='approved'
        )
        
        # Create loan
        Loan.objects.create(
            loan_number=f'PERF-LOAN-{i:06d}',
            application=app,
            borrower=borrower,
            principal_amount=principal,
            interest_amount=interest,
            processing_fee=processing_fee,
            total_amount=total,
            disbursement_date=disbursement_date,
            due_date=due_date,
            duration_days=30,
            status=random.choice(statuses)
        )
        
        if (i + 1) % 100 == 0:
            print(f"  Generated {i + 1} loans...")
    
    print(f"✓ Successfully generated {count} test loans")

def generate_test_applications(count=1000):
    """Generate test applications for age & gender analytics"""
    print(f"Generating {count} test applications...")
    
    # Get or create test data
    branch = Branch.objects.first()
    product = LoanProduct.objects.first()
    
    # Generate borrowers with demographics
    genders = ['M', 'F', 'O', None]
    
    for i in range(count):
        # Random demographics
        age = random.randint(18, 70)
        birth_date = datetime.now().date() - timedelta(days=age*365)
        gender = random.choice(genders)
        
        # Create borrower
        borrower = CustomUser.objects.create_user(
            username=f'perf_borrower_{i:06d}',
            email=f'perf_borrower_{i:06d}@test.com',
            password='testpass123',
            role='borrower',
            branch=branch,
            phone_number=f'+2547{i:08d}',
            date_of_birth=birth_date,
            gender=gender
        )
        
        # Create application
        LoanApplication.objects.create(
            application_number=f'PERF-APP-{i:06d}',
            borrower=borrower,
            loan_product=product,
            requested_amount=Decimal(random.randint(5000, 50000)),
            requested_duration=30,
            purpose='Performance test',
            status=random.choice(['pending', 'approved', 'rejected'])
        )
        
        if (i + 1) % 100 == 0:
            print(f"  Generated {i + 1} applications...")
    
    print(f"✓ Successfully generated {count} test applications")

if __name__ == '__main__':
    import sys
    
    if len(sys.argv) > 1:
        if sys.argv[1] == 'loans':
            count = int(sys.argv[2]) if len(sys.argv) > 2 else 500
            generate_test_loans(count)
        elif sys.argv[1] == 'applications':
            count = int(sys.argv[2]) if len(sys.argv) > 2 else 1000
            generate_test_applications(count)
        else:
            print("Usage: python generate_test_data.py [loans|applications] [count]")
    else:
        # Generate both
        generate_test_loans(500)
        generate_test_applications(1000)
```

### Generate Test Data

```bash
# Generate 500 test loans
python generate_test_data.py loans 500

# Generate 1000 test applications
python generate_test_data.py applications 1000
```

## 1. Disbursed Loans Report Performance

### 1.1 Test Setup
- [ ] Ensure database has 500+ loans
- [ ] Clear any caches
- [ ] Restart development server

### 1.2 Performance Test

```python
# test_disbursed_loans_performance.py
import time
from django.test import Client
from django.contrib.auth import get_user_model

User = get_user_model()

def test_disbursed_loans_report_performance():
    """Test disbursed loans report loads within 3 seconds for 500 loans"""
    
    # Setup
    client = Client()
    admin = User.objects.filter(role='admin').first()
    client.force_login(admin)
    
    # Warm up (first request may be slower due to caching)
    client.get('/reports/disbursed-loans/')
    
    # Actual test
    start_time = time.time()
    response = client.get('/reports/disbursed-loans/')
    end_time = time.time()
    
    duration = end_time - start_time
    
    print(f"Disbursed loans report load time: {duration:.2f} seconds")
    print(f"Requirement: < 3 seconds")
    print(f"Status: {'✓ PASS' if duration < 3 else '✗ FAIL'}")
    
    assert response.status_code == 200
    assert duration < 3.0, f"Report took {duration:.2f}s, exceeds 3s requirement"

if __name__ == '__main__':
    test_disbursed_loans_report_performance()
```

### 1.3 Manual Test
- [ ] Navigate to `/reports/disbursed-loans/`
- [ ] Start timer
- [ ] Wait for page to fully load
- [ ] Stop timer
- [ ] Record time: _______ seconds
- [ ] Result: [ ] Pass (< 3s) / [ ] Fail (>= 3s)

### 1.4 With Filters
- [ ] Apply filters (status, date range, branch)
- [ ] Measure load time: _______ seconds
- [ ] Result: [ ] Pass (< 3s) / [ ] Fail (>= 3s)

### 1.5 With Sorting
- [ ] Sort by different columns
- [ ] Measure load time for each: _______ seconds
- [ ] Result: [ ] Pass (< 3s) / [ ] Fail (>= 3s)

## 2. Age & Gender Analytics Performance

### 2.1 Test Setup
- [ ] Ensure database has 1000+ applications
- [ ] Clear any caches
- [ ] Restart development server

### 2.2 Performance Test

```python
# test_age_gender_analytics_performance.py
import time
from django.test import Client
from django.contrib.auth import get_user_model

User = get_user_model()

def test_age_gender_analytics_performance():
    """Test age & gender analytics calculates within 5 seconds for 1000 applications"""
    
    # Setup
    client = Client()
    admin = User.objects.filter(role='admin').first()
    client.force_login(admin)
    
    # Warm up
    client.get('/reports/age-gender-analytics/')
    
    # Actual test
    start_time = time.time()
    response = client.get('/reports/age-gender-analytics/')
    end_time = time.time()
    
    duration = end_time - start_time
    
    print(f"Age & gender analytics load time: {duration:.2f} seconds")
    print(f"Requirement: < 5 seconds")
    print(f"Status: {'✓ PASS' if duration < 5 else '✗ FAIL'}")
    
    assert response.status_code == 200
    assert duration < 5.0, f"Analytics took {duration:.2f}s, exceeds 5s requirement"

if __name__ == '__main__':
    test_age_gender_analytics_performance()
```

### 2.3 Manual Test
- [ ] Navigate to `/reports/age-gender-analytics/`
- [ ] Start timer
- [ ] Wait for page to fully load (including charts)
- [ ] Stop timer
- [ ] Record time: _______ seconds
- [ ] Result: [ ] Pass (< 5s) / [ ] Fail (>= 5s)

## 3. PDF Export Performance

### 3.1 Test Setup
- [ ] Ensure database has 1000+ records
- [ ] Clear any caches

### 3.2 Performance Test

```python
# test_pdf_export_performance.py
import time
from django.test import Client
from django.contrib.auth import get_user_model

User = get_user_model()

def test_pdf_export_performance():
    """Test PDF export generates within 30 seconds for 1000 records"""
    
    # Setup
    client = Client()
    admin = User.objects.filter(role='admin').first()
    client.force_login(admin)
    
    # Test PDF export
    start_time = time.time()
    response = client.get('/reports/disbursed-loans/?export=pdf')
    end_time = time.time()
    
    duration = end_time - start_time
    
    print(f"PDF export time: {duration:.2f} seconds")
    print(f"Requirement: < 30 seconds")
    print(f"Status: {'✓ PASS' if duration < 30 else '✗ FAIL'}")
    
    assert response.status_code == 200
    assert response['Content-Type'] == 'application/pdf'
    assert duration < 30.0, f"PDF export took {duration:.2f}s, exceeds 30s requirement"

if __name__ == '__main__':
    test_pdf_export_performance()
```

### 3.3 Manual Test
- [ ] Navigate to disbursed loans report
- [ ] Click "Export to PDF"
- [ ] Start timer
- [ ] Wait for PDF to download
- [ ] Stop timer
- [ ] Record time: _______ seconds
- [ ] Result: [ ] Pass (< 30s) / [ ] Fail (>= 30s)

### 3.4 Test with Different Reports
- [ ] Test PDF export for age & gender analytics
  - [ ] Time: _______ seconds
  - [ ] Result: [ ] Pass / [ ] Fail
- [ ] Test PDF export for loans due report
  - [ ] Time: _______ seconds
  - [ ] Result: [ ] Pass / [ ] Fail

## 4. Excel Export Performance

### 4.1 Test Setup
- [ ] Ensure database has 1000+ records
- [ ] Clear any caches

### 4.2 Performance Test

```python
# test_excel_export_performance.py
import time
from django.test import Client
from django.contrib.auth import get_user_model

User = get_user_model()

def test_excel_export_performance():
    """Test Excel export generates within 20 seconds for 1000 records"""
    
    # Setup
    client = Client()
    admin = User.objects.filter(role='admin').first()
    client.force_login(admin)
    
    # Test Excel export
    start_time = time.time()
    response = client.get('/reports/disbursed-loans/?export=excel')
    end_time = time.time()
    
    duration = end_time - start_time
    
    print(f"Excel export time: {duration:.2f} seconds")
    print(f"Requirement: < 20 seconds")
    print(f"Status: {'✓ PASS' if duration < 20 else '✗ FAIL'}")
    
    assert response.status_code == 200
    assert 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet' in response['Content-Type']
    assert duration < 20.0, f"Excel export took {duration:.2f}s, exceeds 20s requirement"

if __name__ == '__main__':
    test_excel_export_performance()
```

### 4.3 Manual Test
- [ ] Navigate to disbursed loans report
- [ ] Click "Export to Excel"
- [ ] Start timer
- [ ] Wait for Excel file to download
- [ ] Stop timer
- [ ] Record time: _______ seconds
- [ ] Result: [ ] Pass (< 20s) / [ ] Fail (>= 20s)

### 4.4 Test with Different Reports
- [ ] Test Excel export for age & gender analytics
  - [ ] Time: _______ seconds
  - [ ] Result: [ ] Pass / [ ] Fail
- [ ] Test Excel export for loans due report
  - [ ] Time: _______ seconds
  - [ ] Result: [ ] Pass / [ ] Fail

## 5. Database Query Performance

### 5.1 Query Analysis

```python
# analyze_queries.py
from django.test.utils import override_settings
from django.db import connection
from django.test import Client
from django.contrib.auth import get_user_model

User = get_user_model()

@override_settings(DEBUG=True)
def analyze_report_queries():
    """Analyze database queries for reports"""
    
    client = Client()
    admin = User.objects.filter(role='admin').first()
    client.force_login(admin)
    
    # Reset query log
    connection.queries_log.clear()
    
    # Make request
    response = client.get('/reports/disbursed-loans/')
    
    # Analyze queries
    queries = connection.queries
    total_time = sum(float(q['time']) for q in queries)
    
    print(f"Total queries: {len(queries)}")
    print(f"Total query time: {total_time:.4f} seconds")
    print(f"Average query time: {total_time/len(queries):.4f} seconds")
    
    # Find slow queries (> 0.1s)
    slow_queries = [q for q in queries if float(q['time']) > 0.1]
    if slow_queries:
        print(f"\nSlow queries (> 0.1s): {len(slow_queries)}")
        for q in slow_queries:
            print(f"  {q['time']}s: {q['sql'][:100]}...")
    
    # Check for N+1 queries
    sql_patterns = {}
    for q in queries:
        pattern = q['sql'].split('WHERE')[0] if 'WHERE' in q['sql'] else q['sql']
        sql_patterns[pattern] = sql_patterns.get(pattern, 0) + 1
    
    repeated_queries = {k: v for k, v in sql_patterns.items() if v > 10}
    if repeated_queries:
        print(f"\nPotential N+1 queries:")
        for pattern, count in repeated_queries.items():
            print(f"  {count}x: {pattern[:100]}...")

if __name__ == '__main__':
    analyze_report_queries()
```

### 5.2 Query Optimization Checks
- [ ] Run query analysis script
- [ ] Check for N+1 queries
- [ ] Check for missing indexes
- [ ] Check for slow queries (> 0.1s)
- [ ] Document findings: _______________

## 6. Concurrent User Testing

### 6.1 Load Testing Script

```python
# load_test.py
import concurrent.futures
import time
from django.test import Client
from django.contrib.auth import get_user_model

User = get_user_model()

def simulate_user_request(user_id):
    """Simulate a single user request"""
    client = Client()
    admin = User.objects.filter(role='admin').first()
    client.force_login(admin)
    
    start_time = time.time()
    response = client.get('/reports/disbursed-loans/')
    end_time = time.time()
    
    return {
        'user_id': user_id,
        'status_code': response.status_code,
        'duration': end_time - start_time
    }

def load_test(num_users=10):
    """Simulate multiple concurrent users"""
    print(f"Simulating {num_users} concurrent users...")
    
    with concurrent.futures.ThreadPoolExecutor(max_workers=num_users) as executor:
        futures = [executor.submit(simulate_user_request, i) for i in range(num_users)]
        results = [f.result() for f in concurrent.futures.as_completed(futures)]
    
    # Analyze results
    durations = [r['duration'] for r in results]
    avg_duration = sum(durations) / len(durations)
    max_duration = max(durations)
    min_duration = min(durations)
    
    print(f"\nResults:")
    print(f"  Average response time: {avg_duration:.2f}s")
    print(f"  Min response time: {min_duration:.2f}s")
    print(f"  Max response time: {max_duration:.2f}s")
    print(f"  All requests successful: {all(r['status_code'] == 200 for r in results)}")

if __name__ == '__main__':
    load_test(10)
```

### 6.2 Concurrent User Test
- [ ] Run load test with 10 concurrent users
- [ ] Average response time: _______ seconds
- [ ] Max response time: _______ seconds
- [ ] All requests successful: [ ] Yes / [ ] No
- [ ] Result: [ ] Pass / [ ] Fail

## 7. Memory Usage Testing

### 7.1 Memory Profiling

```python
# memory_profile.py
import tracemalloc
from django.test import Client
from django.contrib.auth import get_user_model

User = get_user_model()

def profile_memory_usage():
    """Profile memory usage during report generation"""
    
    client = Client()
    admin = User.objects.filter(role='admin').first()
    client.force_login(admin)
    
    # Start memory tracking
    tracemalloc.start()
    
    # Make request
    response = client.get('/reports/disbursed-loans/')
    
    # Get memory usage
    current, peak = tracemalloc.get_traced_memory()
    tracemalloc.stop()
    
    print(f"Current memory usage: {current / 1024 / 1024:.2f} MB")
    print(f"Peak memory usage: {peak / 1024 / 1024:.2f} MB")
    
    # Check for memory leaks
    if peak > 500 * 1024 * 1024:  # 500 MB
        print("⚠ WARNING: High memory usage detected")

if __name__ == '__main__':
    profile_memory_usage()
```

### 7.2 Memory Test
- [ ] Run memory profiling script
- [ ] Current memory usage: _______ MB
- [ ] Peak memory usage: _______ MB
- [ ] Memory leaks detected: [ ] Yes / [ ] No
- [ ] Result: [ ] Pass / [ ] Fail

## Performance Test Results Summary

### Disbursed Loans Report (Requirement: < 3s for 500 loans)
- [ ] Test passed
- [ ] Actual time: _______ seconds
- [ ] With filters: _______ seconds
- [ ] With sorting: _______ seconds
- [ ] Notes: _______________

### Age & Gender Analytics (Requirement: < 5s for 1000 applications)
- [ ] Test passed
- [ ] Actual time: _______ seconds
- [ ] Notes: _______________

### PDF Export (Requirement: < 30s for 1000 records)
- [ ] Test passed
- [ ] Disbursed loans: _______ seconds
- [ ] Age & gender analytics: _______ seconds
- [ ] Loans due report: _______ seconds
- [ ] Notes: _______________

### Excel Export (Requirement: < 20s for 1000 records)
- [ ] Test passed
- [ ] Disbursed loans: _______ seconds
- [ ] Age & gender analytics: _______ seconds
- [ ] Loans due report: _______ seconds
- [ ] Notes: _______________

### Database Queries
- [ ] No N+1 queries detected
- [ ] No slow queries (> 0.1s)
- [ ] Total queries: _______
- [ ] Total query time: _______ seconds
- [ ] Notes: _______________

### Concurrent Users (10 users)
- [ ] Test passed
- [ ] Average response time: _______ seconds
- [ ] Max response time: _______ seconds
- [ ] Notes: _______________

### Memory Usage
- [ ] Test passed
- [ ] Peak memory usage: _______ MB
- [ ] No memory leaks detected
- [ ] Notes: _______________

## Overall Performance Assessment

**Performance Requirements Met:** [ ] Yes / [ ] No  
**Critical Performance Issues:** _______________  
**Optimization Needed:** [ ] Yes / [ ] No  
**Ready for Deployment:** [ ] Yes / ] No  

**Tester Name:** _______________  
**Date:** _______________  
**Signature:** _______________

## Optimization Recommendations

### If Performance Issues Found:

1. **Database Optimization:**
   - Add indexes on frequently queried fields
   - Optimize complex queries
   - Use select_related() and prefetch_related()
   - Consider database query caching

2. **Application Optimization:**
   - Implement pagination for large datasets
   - Use lazy loading for heavy computations
   - Cache expensive calculations
   - Optimize template rendering

3. **Export Optimization:**
   - Stream large exports instead of loading all in memory
   - Use background tasks for very large exports
   - Implement progress indicators
   - Consider chunked processing

4. **Infrastructure:**
   - Increase server resources (CPU, RAM)
   - Use CDN for static assets
   - Implement load balancing
   - Consider database replication

## Next Steps

1. **If all tests pass:**
   - Document performance baselines
   - Set up performance monitoring
   - Proceed to deployment

2. **If performance issues found:**
   - Prioritize optimization efforts
   - Implement optimizations
   - Re-run performance tests
   - Document improvements

3. **Ongoing monitoring:**
   - Set up performance alerts
   - Monitor production performance
   - Regular performance audits
   - Capacity planning