Skip to main content

Performance Optimization

This guide covers performance optimization techniques, monitoring, and troubleshooting for NopeSight deployments.

Dashboard Performance (v3.1.0)​

API Call Optimization​

Starting with v3.1.0, NopeSight includes significant dashboard performance improvements:

Smart Caching System​

dashboard_cache:
  enabled: true
  ttl_seconds: 300  # 5 minutes
  redis_prefix: "nopesight:cache"
  
  cache_strategies:
    widget_data:
      ttl: 300
      invalidation: "tenant_aware"
    
    ci_counts:
      ttl: 600
      invalidation: "on_update"
    
    relationship_stats:
      ttl: 900
      invalidation: "periodic"

API Deduplication​

  • 50-60% Reduction: Dashboard API calls reduced through intelligent deduplication
  • Batch Loading: Widget data loaded in optimized batches
  • Request Coalescing: Multiple identical requests merged into single API call

Performance Metrics​

Monitor dashboard performance through these key indicators:

performance_metrics:
  page_load_time:
    target: < 2_seconds
    acceptable: < 5_seconds
    
  api_response_time:
    target: < 500_milliseconds
    acceptable: < 2_seconds
    
  cache_hit_rate:
    target: > 80%
    minimum: > 60%
    
  concurrent_users:
    supported: 100
    tested_limit: 250

Dashboard Optimization Best Practices​

  1. Widget Configuration:

    • Limit widgets per dashboard (recommended: max 12)
    • Use appropriate data refresh intervals
    • Implement skeleton loading for better UX

  2. Data Filtering:

    • Apply tenant filters early in queries
    • Use indexed fields for filtering
    • Implement pagination for large datasets

  3. Caching Strategy:

    • Cache frequently accessed data
    • Implement cache invalidation on data changes
    • Use Redis for distributed caching

Database Performance​

MongoDB Optimization​

mongodb_optimization:
  connection_pool:
    min_size: 5
    max_size: 50
    max_idle_time: 30000
    
  query_optimization:
    enable_profiling: true
    slow_query_threshold: 100  # milliseconds
    
  indexing:
    compound_indexes:
      - ["tenant", "ciType", "status"]
      - ["tenant", "lastScan"]
      - ["tenant", "customFields.environment"]
    
    text_indexes:
      - ["name", "description"]

Index Monitoring​

# Check index usage
db.cis.aggregate([
  { $indexStats: {} }
])

# Identify slow queries
db.setProfilingLevel(2, { slowms: 100 })
db.system.profile.find().sort({ ts: -1 }).limit(5)

# Monitor query performance
db.cis.explain("executionStats").find({
  tenant: "IT",
  ciType: "Server"
})

Redis Performance​

Queue Optimization​

redis_optimization:
  queues:
    scan_processing:
      concurrency: 5
      batch_size: 10
      retry_attempts: 3
      
    ai_analysis:
      concurrency: 2
      rate_limit: "10/minute"
      timeout: 300000  # 5 minutes
      
    notifications:
      concurrency: 10
      batch_size: 50
      
  memory_management:
    max_memory: "2gb"
    policy: "allkeys-lru"
    
  persistence:
    save: "900 1"  # Save if 1 key changed in 900 seconds
    appendonly: true
    appendfsync: "everysec"

Cache Performance​

# Monitor Redis performance
redis-cli info memory
redis-cli info stats
redis-cli info commandstats

# Check cache hit rates
redis-cli info stats | grep cache

# Monitor slow queries
redis-cli slowlog get 10

Application Performance​

Node.js Optimization​

nodejs_optimization:
  memory:
    max_old_space_size: 4096  # MB
    max_semi_space_size: 512  # MB
    
  cluster:
    enabled: true
    workers: "auto"  # CPU cores
    
  monitoring:
    heap_snapshots: true
    cpu_profiling: true
    
  garbage_collection:
    expose_gc: true
    trace_gc: true

API Performance​

// Rate limiting configuration
const rateLimit = {
  general: {
    windowMs: 60000,    // 1 minute
    max: 100,           // requests per window
  },
  
  ai_endpoints: {
    windowMs: 60000,
    max: 10,            // AI requests per minute
  },
  
  bulk_operations: {
    windowMs: 60000,
    max: 5,             // Bulk operations per minute
  }
};

Monitoring and Alerting​

Performance Monitoring​

monitoring:
  metrics:
    - api_response_time
    - database_query_time
    - cache_hit_rate
    - memory_usage
    - cpu_utilization
    - queue_processing_time
    
  alerting:
    high_api_latency:
      threshold: 2000  # milliseconds
      action: "scale_horizontally"
      
    low_cache_hit_rate:
      threshold: 60    # percentage
      action: "investigate_cache_config"
      
    high_memory_usage:
      threshold: 85    # percentage
      action: "restart_application"

CloudWatch Integration​

cloudwatch:
  enabled: true
  namespace: "NopeSight/Performance"
  
  custom_metrics:
    - name: "DashboardLoadTime"
      unit: "Milliseconds"
      dimensions:
        - name: "Environment"
          value: "${ENVIRONMENT}"
    
    - name: "CacheHitRate"
      unit: "Percent"
      dimensions:
        - name: "CacheType"
          value: "Dashboard"

Troubleshooting Performance Issues​

Common Performance Problems​

  1. Slow Dashboard Loading:

    # Check cache status
    redis-cli info keyspace

    # Verify API response times
    curl -w "@curl-format.txt" -s -o /dev/null https://api.nopesight.com/api/dashboard/stats

    # Monitor database queries
    tail -f /var/log/mongodb/mongod.log | grep slow

  2. High Memory Usage:

    # Check Node.js memory usage
    node --expose-gc app.js

    # Monitor heap snapshots
    curl http://localhost:3000/debug/heap-snapshot > heap.snapshot

    # Check Redis memory
    redis-cli info memory

  3. Queue Backlogs:

    # Monitor queue sizes
    redis-cli llen "bull:scan_processing:waiting"
    redis-cli llen "bull:ai_analysis:waiting"

    # Check failed jobs
    redis-cli llen "bull:scan_processing:failed"

Performance Tuning Steps​

  1. Database Optimization:

    • Add missing indexes
    • Optimize slow queries
    • Enable query profiling
    • Consider read replicas

  2. Cache Optimization:

    • Increase cache TTL for stable data
    • Implement cache warming
    • Monitor cache eviction rates
    • Use appropriate cache keys

  3. Application Scaling:

    • Enable clustering for Node.js
    • Implement horizontal scaling
    • Use load balancers
    • Optimize connection pools

Performance Testing​

Load Testing​

# API load testing with Apache Bench
ab -n 1000 -c 10 -H "Authorization: Bearer TOKEN" \
   https://api.nopesight.com/api/dashboard/stats

# Redis performance testing
redis-benchmark -h localhost -p 6379 -c 50 -n 10000

# Database performance testing
mongo --eval "
  db.cis.find({tenant: 'IT'}).explain('executionStats')
"

Monitoring During Load​

# Monitor system resources
top -p $(pgrep node)
iostat -x 1
free -m

# Monitor application metrics
curl http://localhost:3000/metrics

# Monitor database performance
mongostat --host localhost:27017

Note: Regular performance monitoring and optimization ensure NopeSight maintains optimal response times as your infrastructure data grows.