mirror of
https://github.com/pacnpal/thrilltrack-explorer.git
synced 2025-12-24 01:51:13 -05:00
Add automated data retention cleanup
Implements edge function, Django tasks, and UI hooks/panels for automatic retention of old metrics, anomalies, alerts, and incidents, plus updates to query keys and monitoring dashboard to reflect data-retention workflows.
This commit is contained in:
@@ -136,6 +136,24 @@ SELECT cron.schedule(
|
||||
);
|
||||
```
|
||||
|
||||
### 5. Data Retention Cleanup Setup
|
||||
|
||||
The `data-retention-cleanup` edge function should run daily:
|
||||
|
||||
```sql
|
||||
SELECT cron.schedule(
|
||||
'data-retention-cleanup-daily',
|
||||
'0 3 * * *', -- Daily at 3:00 AM
|
||||
$$
|
||||
SELECT net.http_post(
|
||||
url:='https://api.thrillwiki.com/functions/v1/data-retention-cleanup',
|
||||
headers:='{"Content-Type": "application/json", "Authorization": "Bearer YOUR_ANON_KEY"}'::jsonb,
|
||||
body:=concat('{"time": "', now(), '"}')::jsonb
|
||||
) as request_id;
|
||||
$$
|
||||
);
|
||||
```
|
||||
|
||||
## Metrics Collected
|
||||
|
||||
### Django Metrics
|
||||
@@ -154,6 +172,35 @@ SELECT cron.schedule(
|
||||
- `submission_approval_rate`: Percentage of approved submissions (workflow)
|
||||
- `avg_moderation_time`: Average time to moderate in minutes (workflow)
|
||||
|
||||
## Data Retention Policies
|
||||
|
||||
The system automatically cleans up old data to manage database size:
|
||||
|
||||
### Retention Periods
|
||||
- **Metrics** (`metric_time_series`): 30 days
|
||||
- **Anomaly Detections**: 30 days (resolved alerts archived after 7 days)
|
||||
- **Resolved Alerts**: 90 days
|
||||
- **Resolved Incidents**: 90 days
|
||||
|
||||
### Cleanup Functions
|
||||
|
||||
The following database functions manage data retention:
|
||||
|
||||
1. **`cleanup_old_metrics(retention_days)`**: Deletes metrics older than specified days (default: 30)
|
||||
2. **`cleanup_old_anomalies(retention_days)`**: Archives resolved anomalies and deletes old unresolved ones (default: 30)
|
||||
3. **`cleanup_old_alerts(retention_days)`**: Deletes old resolved alerts (default: 90)
|
||||
4. **`cleanup_old_incidents(retention_days)`**: Deletes old resolved incidents (default: 90)
|
||||
5. **`run_data_retention_cleanup()`**: Master function that runs all cleanup operations
|
||||
|
||||
### Automated Cleanup Schedule
|
||||
|
||||
Django Celery tasks run retention cleanup automatically:
|
||||
- Full cleanup: Daily at 3:00 AM
|
||||
- Metrics cleanup: Daily at 3:30 AM
|
||||
- Anomaly cleanup: Daily at 4:00 AM
|
||||
|
||||
View retention statistics in the Admin Dashboard's Data Retention panel.
|
||||
|
||||
## Monitoring
|
||||
|
||||
View collected metrics in the Admin Monitoring Dashboard:
|
||||
|
||||
Reference in New Issue
Block a user