Software Updates and Maintenance

Maintaining an up-to-date Ridgeback environment ensures you stay ahead of security vulnerabilities, performance issues, and compatibility breaks. This chapter focuses on upgrading service containers and Rcores, as well as managing database and log growth over time through pruning strategies. By incorporating regular maintenance into your routine, you’ll keep Ridgeback stable, secure, and performing optimally.

Upgrading the Service Containers

When to Upgrade:

• New Releases: Ridgeback periodically releases updates that may include security patches, performance optimizations, new features, and bug fixes.
• Support or Compliance: Organizational policies or compliance requirements might mandate keeping software at a supported version.

How to Upgrade:

1. Backup Current State:

   • Export or snapshot the .env and docker-compose.yml files.
   • Ensure you have recent database backups in case a rollback is needed.

2. Update Versions:

   • Edit the image tags in your docker-compose.yml file to be the version number you want to upgrade to. For example, server:3.0.0 might be changed to server:3.1.0. If you want to always stay up to date with the latest version 3, then use the latest string in the version, like this: server:3.latest.

3. Pull the Latest Images:

   docker compose pull
   

   This updates all referenced images to their newest versions available in the registry.

4. Recreate the Containers:

   docker compose up -d
   

   The up -d command with updated images will recreate containers using the new versions.

5. Check Logs and Status:

   docker compose logs -f
   

   Verify no startup errors appear and confirm the Ridgeback UI is operational.

6. Post-Upgrade Testing:

   • Log into the UI and confirm all services are running and accessible.
   • Review a few policies, events, or reports to ensure functionality is intact.

Rollback Plan: If a new version introduces issues, stop the updated containers and revert to the previous known-good images:

docker compose down
docker compose up -d --build --no-cache

Use previously saved images (tagged or saved via docker save) or rely on your backups.

Upgrading the Rcores

When to Upgrade:

• New Rcore Binaries Released: Periodic Rcore releases improve packet handling, phantom logic, or introduce performance fixes.
• Feature Requirements: Some new policies or phantom modes require newer Rcore versions.

How to Upgrade:

1. Download the Latest Rcore Executable:
   Obtain the new binary from the Ridgeback vendor portal or your internal repository.

2. Stop the Existing Rcore:

   systemctl stop rcore
   

   or, if running manually, Ctrl+C or kill the process.

3. Replace the Executable:

   cp rcore-linux-newversion /usr/local/bin/rcore-linux
   chmod +x /usr/local/bin/rcore-linux
   

   On Windows, replace rcore-win.exe in the designated directory.

4. Restart the Rcore:

   systemctl start rcore
   

   or run the startup script again.

5. Validate Operation:

   • Check logs to ensure the Rcore connects to the manager and reports events.
   • Confirm the Rcore status in the Ridgeback UI.

Rollback Plan: Keep a copy of the old Rcore binary. If the new one causes issues, revert by stopping the Rcore, restoring the old binary, and restarting.

Pruning the Databases and Logs

Over time, Ridgeback accumulates a large volume of network events and system logs. Pruning old data helps maintain database performance, reduces storage costs, and keeps logs manageable.

Pruning the Database

Why Prune:

• Performance: Huge NetEvent tables slow down queries and indexing.
• Cost: Cloud databases charge by storage; smaller datasets mean lower costs.
• Compliance: Some regulations require deleting data after a certain retention period.

How to Prune:

1. Establish a Retention Policy:
   Decide how long to keep historical events—e.g., 90 days or 6 months.

2. Use SQL Queries to Delete Old Data:

   DELETE FROM NetEvent
   WHERE time < NOW() - INTERVAL 90 DAY;
   

   Run this during low-traffic hours. Consider batching deletions to avoid large performance hits.

3. Partitioning or Archiving:

   • Consider using MySQL partitioning to manage data by date range, making pruning a matter of dropping old partitions.
   • Export older data to CSV or another format for offline archival before deletion if needed.

4. Database Maintenance Tasks:

   • After pruning, run OPTIMIZE TABLE or ANALYZE TABLE to improve performance.
   • Monitor free disk space and confirm improvements.

Pruning the Logs

Service Container Logs:

• Docker Log Rotation:
  Configure log rotation in docker-compose.yml:

  logging:
    driver: "json-file"
    options:
      max-size: "10m"
      max-file: "3"
  

  This automatically rotates container logs, keeping them within manageable sizes.

• External Logging:
  If using centralized logging (e.g., ELK or Splunk), apply retention policies or index lifecycles to prune old logs.

Rcore Logs:

• If Rcore logs are redirected to a file, use system tools like logrotate on Linux:

  sudo nano /etc/logrotate.d/rcore
  

  Configure a rotation schedule, e.g., rotate weekly and keep 4 weeks of logs.

Review Retention Policies Regularly:

• Adjust retention based on audit requirements, storage costs, and operational needs.
• Consider alerts to warn when logs or database tables approach size limits.

Best Practices for Maintenance

1. Routine Backups and Tests:

   • Before any major upgrade or pruning operation, ensure you have recent, test-restored backups.
   • Regularly test backup restores in a dev environment to confirm data integrity.

2. Scheduled Maintenance Windows:

   • Perform upgrades, pruning, and heavy maintenance tasks during planned maintenance windows to minimize user impact.
   • Notify stakeholders in advance.

3. Monitoring and Alerts:

   • Set up alerts if the database grows too large or if logs are not rotating as expected.
   • Monitor performance metrics like query response times or Rcore CPU usage after upgrades or pruning tasks.

4. Document Procedures:

   • Keep a runbook or SOPs for upgrades, rollbacks, and pruning steps.
   • Ensure multiple team members know how to execute these procedures.