Add automated data retention cleanup

Implements edge function, Django tasks, and UI hooks/panels for automatic retention of old metrics, anomalies, alerts, and incidents, plus updates to query keys and monitoring dashboard to reflect data-retention workflows.
This commit is contained in:
gpt-engineer-app[bot]
2025-11-11 02:21:27 +00:00
parent 07fdfe34f3
commit 915a9fe2df
9 changed files with 589 additions and 0 deletions

View File

@@ -136,6 +136,24 @@ SELECT cron.schedule(
);
```
### 5. Data Retention Cleanup Setup
The `data-retention-cleanup` edge function should run daily:
```sql
SELECT cron.schedule(
'data-retention-cleanup-daily',
'0 3 * * *', -- Daily at 3:00 AM
$$
SELECT net.http_post(
url:='https://api.thrillwiki.com/functions/v1/data-retention-cleanup',
headers:='{"Content-Type": "application/json", "Authorization": "Bearer YOUR_ANON_KEY"}'::jsonb,
body:=concat('{"time": "', now(), '"}')::jsonb
) as request_id;
$$
);
```
## Metrics Collected
### Django Metrics
@@ -154,6 +172,35 @@ SELECT cron.schedule(
- `submission_approval_rate`: Percentage of approved submissions (workflow)
- `avg_moderation_time`: Average time to moderate in minutes (workflow)
## Data Retention Policies
The system automatically cleans up old data to manage database size:
### Retention Periods
- **Metrics** (`metric_time_series`): 30 days
- **Anomaly Detections**: 30 days (resolved alerts archived after 7 days)
- **Resolved Alerts**: 90 days
- **Resolved Incidents**: 90 days
### Cleanup Functions
The following database functions manage data retention:
1. **`cleanup_old_metrics(retention_days)`**: Deletes metrics older than specified days (default: 30)
2. **`cleanup_old_anomalies(retention_days)`**: Archives resolved anomalies and deletes old unresolved ones (default: 30)
3. **`cleanup_old_alerts(retention_days)`**: Deletes old resolved alerts (default: 90)
4. **`cleanup_old_incidents(retention_days)`**: Deletes old resolved incidents (default: 90)
5. **`run_data_retention_cleanup()`**: Master function that runs all cleanup operations
### Automated Cleanup Schedule
Django Celery tasks run retention cleanup automatically:
- Full cleanup: Daily at 3:00 AM
- Metrics cleanup: Daily at 3:30 AM
- Anomaly cleanup: Daily at 4:00 AM
View retention statistics in the Admin Dashboard's Data Retention panel.
## Monitoring
View collected metrics in the Admin Monitoring Dashboard: