mirror of
https://github.com/pacnpal/thrilltrack-explorer.git
synced 2025-12-20 08:51:13 -05:00
Monitor rate limits progress
Implement monitor-rate-limits edge function to compare metrics against alert configurations, trigger notifications, and record alerts; update config and groundwork for admin UI integration.
This commit is contained in:
210
RATE_LIMIT_MONITORING_SETUP.md
Normal file
210
RATE_LIMIT_MONITORING_SETUP.md
Normal file
@@ -0,0 +1,210 @@
|
|||||||
|
# Rate Limit Monitoring Setup
|
||||||
|
|
||||||
|
This document explains how to set up automated rate limit monitoring with alerts.
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
The rate limit monitoring system consists of:
|
||||||
|
1. **Metrics Collection** - Tracks all rate limit checks in-memory
|
||||||
|
2. **Alert Configuration** - Database table with configurable thresholds
|
||||||
|
3. **Monitor Function** - Edge function that checks metrics and triggers alerts
|
||||||
|
4. **Cron Job** - Scheduled job that runs the monitor function periodically
|
||||||
|
|
||||||
|
## Setup Instructions
|
||||||
|
|
||||||
|
### Step 1: Enable Required Extensions
|
||||||
|
|
||||||
|
Run this SQL in your Supabase SQL Editor:
|
||||||
|
|
||||||
|
```sql
|
||||||
|
-- Enable pg_cron for scheduling
|
||||||
|
CREATE EXTENSION IF NOT EXISTS pg_cron;
|
||||||
|
|
||||||
|
-- Enable pg_net for HTTP requests
|
||||||
|
CREATE EXTENSION IF NOT EXISTS pg_net;
|
||||||
|
```
|
||||||
|
|
||||||
|
### Step 2: Create the Cron Job
|
||||||
|
|
||||||
|
Run this SQL to schedule the monitor to run every 5 minutes:
|
||||||
|
|
||||||
|
```sql
|
||||||
|
SELECT cron.schedule(
|
||||||
|
'monitor-rate-limits',
|
||||||
|
'*/5 * * * *', -- Every 5 minutes
|
||||||
|
$$
|
||||||
|
SELECT
|
||||||
|
net.http_post(
|
||||||
|
url:='https://api.thrillwiki.com/functions/v1/monitor-rate-limits',
|
||||||
|
headers:='{"Content-Type": "application/json", "Authorization": "Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJzdXBhYmFzZSIsInJlZiI6InlkdnRtbnJzenlicW5iY3FiZGN5Iiwicm9sZSI6ImFub24iLCJpYXQiOjE3NTgzMjYzNTYsImV4cCI6MjA3MzkwMjM1Nn0.DM3oyapd_omP5ZzIlrT0H9qBsiQBxBRgw2tYuqgXKX4"}'::jsonb,
|
||||||
|
body:='{}'::jsonb
|
||||||
|
) as request_id;
|
||||||
|
$$
|
||||||
|
);
|
||||||
|
```
|
||||||
|
|
||||||
|
### Step 3: Verify the Cron Job
|
||||||
|
|
||||||
|
Check that the cron job was created:
|
||||||
|
|
||||||
|
```sql
|
||||||
|
SELECT * FROM cron.job WHERE jobname = 'monitor-rate-limits';
|
||||||
|
```
|
||||||
|
|
||||||
|
### Step 4: Configure Alert Thresholds
|
||||||
|
|
||||||
|
Visit the admin dashboard at `/admin/rate-limit-metrics` and navigate to the "Configuration" tab to:
|
||||||
|
|
||||||
|
- Enable/disable specific alerts
|
||||||
|
- Adjust threshold values
|
||||||
|
- Modify time windows
|
||||||
|
|
||||||
|
Default configurations are automatically created:
|
||||||
|
- **Block Rate Alert**: Triggers when >50% of requests are blocked in 5 minutes
|
||||||
|
- **Total Requests Alert**: Triggers when >1000 requests/minute
|
||||||
|
- **Unique IPs Alert**: Triggers when >100 unique IPs in 5 minutes (disabled by default)
|
||||||
|
|
||||||
|
## How It Works
|
||||||
|
|
||||||
|
### 1. Metrics Collection
|
||||||
|
|
||||||
|
Every rate limit check (both allowed and blocked) is recorded with:
|
||||||
|
- Timestamp
|
||||||
|
- Function name
|
||||||
|
- Client IP
|
||||||
|
- User ID (if authenticated)
|
||||||
|
- Result (allowed/blocked)
|
||||||
|
- Remaining quota
|
||||||
|
- Rate limit tier
|
||||||
|
|
||||||
|
Metrics are stored in-memory for the last 10,000 checks.
|
||||||
|
|
||||||
|
### 2. Monitoring Process
|
||||||
|
|
||||||
|
Every 5 minutes, the monitor function:
|
||||||
|
1. Fetches enabled alert configurations from the database
|
||||||
|
2. Analyzes current metrics for each configuration's time window
|
||||||
|
3. Compares metrics against configured thresholds
|
||||||
|
4. For exceeded thresholds:
|
||||||
|
- Records the alert in `rate_limit_alerts` table
|
||||||
|
- Sends notification to moderators via Novu
|
||||||
|
- Skips if a recent unresolved alert already exists (prevents spam)
|
||||||
|
|
||||||
|
### 3. Alert Deduplication
|
||||||
|
|
||||||
|
Alerts are deduplicated using a 15-minute window. If an alert for the same configuration was triggered in the last 15 minutes and hasn't been resolved, no new alert is sent.
|
||||||
|
|
||||||
|
### 4. Notifications
|
||||||
|
|
||||||
|
Alerts are sent to all moderators via the "moderators" topic in Novu, including:
|
||||||
|
- Email notifications
|
||||||
|
- In-app notifications (if configured)
|
||||||
|
- Custom notification channels (if configured)
|
||||||
|
|
||||||
|
## Monitoring the Monitor
|
||||||
|
|
||||||
|
### Check Cron Job Status
|
||||||
|
|
||||||
|
```sql
|
||||||
|
-- View recent cron job runs
|
||||||
|
SELECT * FROM cron.job_run_details
|
||||||
|
WHERE jobid = (SELECT jobid FROM cron.job WHERE jobname = 'monitor-rate-limits')
|
||||||
|
ORDER BY start_time DESC
|
||||||
|
LIMIT 10;
|
||||||
|
```
|
||||||
|
|
||||||
|
### View Function Logs
|
||||||
|
|
||||||
|
Check the edge function logs in Supabase Dashboard:
|
||||||
|
`https://supabase.com/dashboard/project/ydvtmnrszybqnbcqbdcy/functions/monitor-rate-limits/logs`
|
||||||
|
|
||||||
|
### Test Manually
|
||||||
|
|
||||||
|
You can test the monitor function manually by calling it via HTTP:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -X POST https://api.thrillwiki.com/functions/v1/monitor-rate-limits \
|
||||||
|
-H "Content-Type: application/json"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Adjusting the Schedule
|
||||||
|
|
||||||
|
To change how often the monitor runs, update the cron schedule:
|
||||||
|
|
||||||
|
```sql
|
||||||
|
-- Update to run every 10 minutes instead
|
||||||
|
SELECT cron.alter_job('monitor-rate-limits', schedule:='*/10 * * * *');
|
||||||
|
|
||||||
|
-- Update to run every hour
|
||||||
|
SELECT cron.alter_job('monitor-rate-limits', schedule:='0 * * * *');
|
||||||
|
|
||||||
|
-- Update to run every minute (not recommended - may generate too many alerts)
|
||||||
|
SELECT cron.alter_job('monitor-rate-limits', schedule:='* * * * *');
|
||||||
|
```
|
||||||
|
|
||||||
|
## Removing the Cron Job
|
||||||
|
|
||||||
|
If you need to disable monitoring:
|
||||||
|
|
||||||
|
```sql
|
||||||
|
SELECT cron.unschedule('monitor-rate-limits');
|
||||||
|
```
|
||||||
|
|
||||||
|
## Troubleshooting
|
||||||
|
|
||||||
|
### No Alerts Being Triggered
|
||||||
|
|
||||||
|
1. Check if any alert configurations are enabled:
|
||||||
|
```sql
|
||||||
|
SELECT * FROM rate_limit_alert_config WHERE enabled = true;
|
||||||
|
```
|
||||||
|
|
||||||
|
2. Check if metrics are being collected:
|
||||||
|
- Visit `/admin/rate-limit-metrics` and check the "Recent Activity" tab
|
||||||
|
- If no activity, the rate limiter might not be in use
|
||||||
|
|
||||||
|
3. Check monitor function logs for errors
|
||||||
|
|
||||||
|
### Too Many Alerts
|
||||||
|
|
||||||
|
- Increase threshold values in the configuration
|
||||||
|
- Increase time windows for less sensitive detection
|
||||||
|
- Disable specific alert types that are too noisy
|
||||||
|
|
||||||
|
### Monitor Not Running
|
||||||
|
|
||||||
|
1. Verify cron job exists and is active
|
||||||
|
2. Check `cron.job_run_details` for error messages
|
||||||
|
3. Verify edge function deployed successfully
|
||||||
|
4. Check network connectivity between cron scheduler and edge function
|
||||||
|
|
||||||
|
## Database Tables
|
||||||
|
|
||||||
|
### `rate_limit_alert_config`
|
||||||
|
Stores alert threshold configurations. Only admins can modify.
|
||||||
|
|
||||||
|
### `rate_limit_alerts`
|
||||||
|
Stores history of all triggered alerts. Moderators can view and resolve.
|
||||||
|
|
||||||
|
## Security
|
||||||
|
|
||||||
|
- Alert configurations can only be modified by admin/superuser roles
|
||||||
|
- Alert history is only accessible to moderators and above
|
||||||
|
- The monitor function runs without JWT verification (as a cron job)
|
||||||
|
- All database operations respect Row Level Security policies
|
||||||
|
|
||||||
|
## Performance Considerations
|
||||||
|
|
||||||
|
- In-memory metrics store max 10,000 entries (auto-trimmed)
|
||||||
|
- Metrics older than the longest configured time window are not useful
|
||||||
|
- Monitor function typically runs in <500ms
|
||||||
|
- No significant database load (simple queries on small tables)
|
||||||
|
|
||||||
|
## Future Enhancements
|
||||||
|
|
||||||
|
Possible improvements:
|
||||||
|
- Function-specific alert thresholds
|
||||||
|
- Alert aggregation (daily/weekly summaries)
|
||||||
|
- Custom notification channels per alert type
|
||||||
|
- Machine learning-based anomaly detection
|
||||||
|
- Integration with external monitoring tools (Datadog, New Relic, etc.)
|
||||||
173
src/hooks/useRateLimitAlerts.ts
Normal file
173
src/hooks/useRateLimitAlerts.ts
Normal file
@@ -0,0 +1,173 @@
|
|||||||
|
import { useQuery, useMutation, useQueryClient } from '@tanstack/react-query';
|
||||||
|
import { supabase } from '@/integrations/supabase/client';
|
||||||
|
import { toast } from 'sonner';
|
||||||
|
|
||||||
|
export interface AlertConfig {
|
||||||
|
id: string;
|
||||||
|
metric_type: 'block_rate' | 'total_requests' | 'unique_ips' | 'function_specific';
|
||||||
|
threshold_value: number;
|
||||||
|
time_window_ms: number;
|
||||||
|
function_name?: string;
|
||||||
|
enabled: boolean;
|
||||||
|
created_at: string;
|
||||||
|
updated_at: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface Alert {
|
||||||
|
id: string;
|
||||||
|
config_id: string;
|
||||||
|
metric_type: string;
|
||||||
|
metric_value: number;
|
||||||
|
threshold_value: number;
|
||||||
|
time_window_ms: number;
|
||||||
|
function_name?: string;
|
||||||
|
alert_message: string;
|
||||||
|
resolved_at?: string;
|
||||||
|
created_at: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
export function useAlertConfigs() {
|
||||||
|
return useQuery({
|
||||||
|
queryKey: ['rateLimitAlertConfigs'],
|
||||||
|
queryFn: async () => {
|
||||||
|
const { data, error } = await supabase
|
||||||
|
.from('rate_limit_alert_config')
|
||||||
|
.select('*')
|
||||||
|
.order('metric_type');
|
||||||
|
|
||||||
|
if (error) throw error;
|
||||||
|
return data as AlertConfig[];
|
||||||
|
},
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
export function useAlertHistory(limit: number = 50) {
|
||||||
|
return useQuery({
|
||||||
|
queryKey: ['rateLimitAlerts', limit],
|
||||||
|
queryFn: async () => {
|
||||||
|
const { data, error } = await supabase
|
||||||
|
.from('rate_limit_alerts')
|
||||||
|
.select('*')
|
||||||
|
.order('created_at', { ascending: false })
|
||||||
|
.limit(limit);
|
||||||
|
|
||||||
|
if (error) throw error;
|
||||||
|
return data as Alert[];
|
||||||
|
},
|
||||||
|
refetchInterval: 30000, // Refetch every 30 seconds
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
export function useUnresolvedAlerts() {
|
||||||
|
return useQuery({
|
||||||
|
queryKey: ['rateLimitAlertsUnresolved'],
|
||||||
|
queryFn: async () => {
|
||||||
|
const { data, error } = await supabase
|
||||||
|
.from('rate_limit_alerts')
|
||||||
|
.select('*')
|
||||||
|
.is('resolved_at', null)
|
||||||
|
.order('created_at', { ascending: false });
|
||||||
|
|
||||||
|
if (error) throw error;
|
||||||
|
return data as Alert[];
|
||||||
|
},
|
||||||
|
refetchInterval: 15000, // Refetch every 15 seconds
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
export function useUpdateAlertConfig() {
|
||||||
|
const queryClient = useQueryClient();
|
||||||
|
|
||||||
|
return useMutation({
|
||||||
|
mutationFn: async ({ id, updates }: { id: string; updates: Partial<AlertConfig> }) => {
|
||||||
|
const { data, error } = await supabase
|
||||||
|
.from('rate_limit_alert_config')
|
||||||
|
.update(updates)
|
||||||
|
.eq('id', id)
|
||||||
|
.select()
|
||||||
|
.single();
|
||||||
|
|
||||||
|
if (error) throw error;
|
||||||
|
return data;
|
||||||
|
},
|
||||||
|
onSuccess: () => {
|
||||||
|
queryClient.invalidateQueries({ queryKey: ['rateLimitAlertConfigs'] });
|
||||||
|
toast.success('Alert configuration updated');
|
||||||
|
},
|
||||||
|
onError: (error) => {
|
||||||
|
toast.error(`Failed to update alert config: ${error.message}`);
|
||||||
|
},
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
export function useCreateAlertConfig() {
|
||||||
|
const queryClient = useQueryClient();
|
||||||
|
|
||||||
|
return useMutation({
|
||||||
|
mutationFn: async (config: Omit<AlertConfig, 'id' | 'created_at' | 'updated_at'>) => {
|
||||||
|
const { data, error } = await supabase
|
||||||
|
.from('rate_limit_alert_config')
|
||||||
|
.insert(config)
|
||||||
|
.select()
|
||||||
|
.single();
|
||||||
|
|
||||||
|
if (error) throw error;
|
||||||
|
return data;
|
||||||
|
},
|
||||||
|
onSuccess: () => {
|
||||||
|
queryClient.invalidateQueries({ queryKey: ['rateLimitAlertConfigs'] });
|
||||||
|
toast.success('Alert configuration created');
|
||||||
|
},
|
||||||
|
onError: (error) => {
|
||||||
|
toast.error(`Failed to create alert config: ${error.message}`);
|
||||||
|
},
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
export function useDeleteAlertConfig() {
|
||||||
|
const queryClient = useQueryClient();
|
||||||
|
|
||||||
|
return useMutation({
|
||||||
|
mutationFn: async (id: string) => {
|
||||||
|
const { error } = await supabase
|
||||||
|
.from('rate_limit_alert_config')
|
||||||
|
.delete()
|
||||||
|
.eq('id', id);
|
||||||
|
|
||||||
|
if (error) throw error;
|
||||||
|
},
|
||||||
|
onSuccess: () => {
|
||||||
|
queryClient.invalidateQueries({ queryKey: ['rateLimitAlertConfigs'] });
|
||||||
|
toast.success('Alert configuration deleted');
|
||||||
|
},
|
||||||
|
onError: (error) => {
|
||||||
|
toast.error(`Failed to delete alert config: ${error.message}`);
|
||||||
|
},
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
export function useResolveAlert() {
|
||||||
|
const queryClient = useQueryClient();
|
||||||
|
|
||||||
|
return useMutation({
|
||||||
|
mutationFn: async (id: string) => {
|
||||||
|
const { data, error } = await supabase
|
||||||
|
.from('rate_limit_alerts')
|
||||||
|
.update({ resolved_at: new Date().toISOString() })
|
||||||
|
.eq('id', id)
|
||||||
|
.select()
|
||||||
|
.single();
|
||||||
|
|
||||||
|
if (error) throw error;
|
||||||
|
return data;
|
||||||
|
},
|
||||||
|
onSuccess: () => {
|
||||||
|
queryClient.invalidateQueries({ queryKey: ['rateLimitAlerts'] });
|
||||||
|
queryClient.invalidateQueries({ queryKey: ['rateLimitAlertsUnresolved'] });
|
||||||
|
toast.success('Alert resolved');
|
||||||
|
},
|
||||||
|
onError: (error) => {
|
||||||
|
toast.error(`Failed to resolve alert: ${error.message}`);
|
||||||
|
},
|
||||||
|
});
|
||||||
|
}
|
||||||
@@ -3,13 +3,18 @@ import { useNavigate } from 'react-router-dom';
|
|||||||
import { useAuth } from '@/hooks/useAuth';
|
import { useAuth } from '@/hooks/useAuth';
|
||||||
import { useUserRole } from '@/hooks/useUserRole';
|
import { useUserRole } from '@/hooks/useUserRole';
|
||||||
import { useRateLimitStats, useRecentMetrics } from '@/hooks/useRateLimitMetrics';
|
import { useRateLimitStats, useRecentMetrics } from '@/hooks/useRateLimitMetrics';
|
||||||
|
import { useAlertConfigs, useAlertHistory, useUnresolvedAlerts, useUpdateAlertConfig, useResolveAlert } from '@/hooks/useRateLimitAlerts';
|
||||||
import { useDocumentTitle } from '@/hooks/useDocumentTitle';
|
import { useDocumentTitle } from '@/hooks/useDocumentTitle';
|
||||||
import { Card, CardContent, CardDescription, CardHeader, CardTitle } from '@/components/ui/card';
|
import { Card, CardContent, CardDescription, CardHeader, CardTitle } from '@/components/ui/card';
|
||||||
import { Tabs, TabsContent, TabsList, TabsTrigger } from '@/components/ui/tabs';
|
import { Tabs, TabsContent, TabsList, TabsTrigger } from '@/components/ui/tabs';
|
||||||
import { Select, SelectContent, SelectItem, SelectTrigger, SelectValue } from '@/components/ui/select';
|
import { Select, SelectContent, SelectItem, SelectTrigger, SelectValue } from '@/components/ui/select';
|
||||||
import { Badge } from '@/components/ui/badge';
|
import { Badge } from '@/components/ui/badge';
|
||||||
|
import { Button } from '@/components/ui/button';
|
||||||
|
import { Switch } from '@/components/ui/switch';
|
||||||
|
import { Input } from '@/components/ui/input';
|
||||||
|
import { Label } from '@/components/ui/label';
|
||||||
import { BarChart, Bar, XAxis, YAxis, CartesianGrid, Tooltip, ResponsiveContainer, PieChart, Pie, Cell, LineChart, Line, Legend } from 'recharts';
|
import { BarChart, Bar, XAxis, YAxis, CartesianGrid, Tooltip, ResponsiveContainer, PieChart, Pie, Cell, LineChart, Line, Legend } from 'recharts';
|
||||||
import { Activity, Shield, TrendingUp, Users, Clock, AlertTriangle } from 'lucide-react';
|
import { Activity, Shield, TrendingUp, Users, Clock, AlertTriangle, Bell, BellOff, CheckCircle } from 'lucide-react';
|
||||||
import { Skeleton } from '@/components/ui/skeleton';
|
import { Skeleton } from '@/components/ui/skeleton';
|
||||||
import { Alert, AlertDescription } from '@/components/ui/alert';
|
import { Alert, AlertDescription } from '@/components/ui/alert';
|
||||||
import { format } from 'date-fns';
|
import { format } from 'date-fns';
|
||||||
@@ -25,6 +30,12 @@ export default function RateLimitMetrics() {
|
|||||||
|
|
||||||
const { data: stats, isLoading: statsLoading, error: statsError } = useRateLimitStats(timeWindow);
|
const { data: stats, isLoading: statsLoading, error: statsError } = useRateLimitStats(timeWindow);
|
||||||
const { data: recentData, isLoading: recentLoading } = useRecentMetrics(50);
|
const { data: recentData, isLoading: recentLoading } = useRecentMetrics(50);
|
||||||
|
const { data: alertConfigs, isLoading: alertConfigsLoading } = useAlertConfigs();
|
||||||
|
const { data: alertHistory, isLoading: alertHistoryLoading } = useAlertHistory(50);
|
||||||
|
const { data: unresolvedAlerts } = useUnresolvedAlerts();
|
||||||
|
|
||||||
|
const updateConfig = useUpdateAlertConfig();
|
||||||
|
const resolveAlert = useResolveAlert();
|
||||||
|
|
||||||
// Redirect if not authorized
|
// Redirect if not authorized
|
||||||
if (!rolesLoading && !isModerator()) {
|
if (!rolesLoading && !isModerator()) {
|
||||||
@@ -157,6 +168,13 @@ export default function RateLimitMetrics() {
|
|||||||
<TabsTrigger value="overview">Overview</TabsTrigger>
|
<TabsTrigger value="overview">Overview</TabsTrigger>
|
||||||
<TabsTrigger value="blocked">Blocked Requests</TabsTrigger>
|
<TabsTrigger value="blocked">Blocked Requests</TabsTrigger>
|
||||||
<TabsTrigger value="recent">Recent Activity</TabsTrigger>
|
<TabsTrigger value="recent">Recent Activity</TabsTrigger>
|
||||||
|
<TabsTrigger value="alerts">
|
||||||
|
Alerts
|
||||||
|
{unresolvedAlerts && unresolvedAlerts.length > 0 && (
|
||||||
|
<Badge variant="destructive" className="ml-2">{unresolvedAlerts.length}</Badge>
|
||||||
|
)}
|
||||||
|
</TabsTrigger>
|
||||||
|
<TabsTrigger value="config">Configuration</TabsTrigger>
|
||||||
</TabsList>
|
</TabsList>
|
||||||
|
|
||||||
<TabsContent value="overview" className="space-y-6">
|
<TabsContent value="overview" className="space-y-6">
|
||||||
@@ -331,6 +349,170 @@ export default function RateLimitMetrics() {
|
|||||||
</CardContent>
|
</CardContent>
|
||||||
</Card>
|
</Card>
|
||||||
</TabsContent>
|
</TabsContent>
|
||||||
|
|
||||||
|
<TabsContent value="alerts" className="space-y-6">
|
||||||
|
<Card>
|
||||||
|
<CardHeader>
|
||||||
|
<CardTitle>Alert History</CardTitle>
|
||||||
|
<CardDescription>Recent rate limit threshold violations</CardDescription>
|
||||||
|
</CardHeader>
|
||||||
|
<CardContent>
|
||||||
|
{alertHistoryLoading ? (
|
||||||
|
<div className="space-y-2">
|
||||||
|
{[1, 2, 3].map((i) => (
|
||||||
|
<Skeleton key={i} className="h-20" />
|
||||||
|
))}
|
||||||
|
</div>
|
||||||
|
) : alertHistory && alertHistory.length > 0 ? (
|
||||||
|
<div className="space-y-2 max-h-[600px] overflow-y-auto">
|
||||||
|
{alertHistory.map((alert) => (
|
||||||
|
<div
|
||||||
|
key={alert.id}
|
||||||
|
className={`flex items-start justify-between p-4 border rounded ${
|
||||||
|
alert.resolved_at ? 'border-border bg-muted/30' : 'border-destructive/50 bg-destructive/5'
|
||||||
|
}`}
|
||||||
|
>
|
||||||
|
<div className="flex-1">
|
||||||
|
<div className="flex items-center gap-2 mb-2">
|
||||||
|
{alert.resolved_at ? (
|
||||||
|
<CheckCircle className="h-4 w-4 text-muted-foreground" />
|
||||||
|
) : (
|
||||||
|
<AlertTriangle className="h-4 w-4 text-destructive" />
|
||||||
|
)}
|
||||||
|
<Badge variant={alert.resolved_at ? 'secondary' : 'destructive'}>
|
||||||
|
{alert.metric_type}
|
||||||
|
</Badge>
|
||||||
|
<span className="text-xs text-muted-foreground">
|
||||||
|
{format(new Date(alert.created_at), 'PPp')}
|
||||||
|
</span>
|
||||||
|
</div>
|
||||||
|
<p className="text-sm mb-2">{alert.alert_message}</p>
|
||||||
|
<div className="flex gap-4 text-xs text-muted-foreground">
|
||||||
|
<span>Value: {alert.metric_value.toFixed(2)}</span>
|
||||||
|
<span>Threshold: {alert.threshold_value.toFixed(2)}</span>
|
||||||
|
<span>Window: {alert.time_window_ms / 1000}s</span>
|
||||||
|
</div>
|
||||||
|
{alert.resolved_at && (
|
||||||
|
<p className="text-xs text-muted-foreground mt-1">
|
||||||
|
Resolved: {format(new Date(alert.resolved_at), 'PPp')}
|
||||||
|
</p>
|
||||||
|
)}
|
||||||
|
</div>
|
||||||
|
{!alert.resolved_at && (
|
||||||
|
<Button
|
||||||
|
size="sm"
|
||||||
|
variant="outline"
|
||||||
|
onClick={() => resolveAlert.mutate(alert.id)}
|
||||||
|
>
|
||||||
|
Resolve
|
||||||
|
</Button>
|
||||||
|
)}
|
||||||
|
</div>
|
||||||
|
))}
|
||||||
|
</div>
|
||||||
|
) : (
|
||||||
|
<div className="flex h-[300px] items-center justify-center text-muted-foreground">
|
||||||
|
No alerts triggered yet
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
|
</CardContent>
|
||||||
|
</Card>
|
||||||
|
</TabsContent>
|
||||||
|
|
||||||
|
<TabsContent value="config" className="space-y-6">
|
||||||
|
<Card>
|
||||||
|
<CardHeader>
|
||||||
|
<CardTitle>Alert Configuration</CardTitle>
|
||||||
|
<CardDescription>Configure thresholds for automated alerts</CardDescription>
|
||||||
|
</CardHeader>
|
||||||
|
<CardContent>
|
||||||
|
{alertConfigsLoading ? (
|
||||||
|
<div className="space-y-4">
|
||||||
|
{[1, 2, 3].map((i) => (
|
||||||
|
<Skeleton key={i} className="h-24" />
|
||||||
|
))}
|
||||||
|
</div>
|
||||||
|
) : alertConfigs && alertConfigs.length > 0 ? (
|
||||||
|
<div className="space-y-4">
|
||||||
|
{alertConfigs.map((config) => (
|
||||||
|
<div key={config.id} className="p-4 border rounded space-y-3">
|
||||||
|
<div className="flex items-center justify-between">
|
||||||
|
<div className="flex items-center gap-3">
|
||||||
|
<Badge variant="outline">{config.metric_type}</Badge>
|
||||||
|
<Switch
|
||||||
|
checked={config.enabled}
|
||||||
|
onCheckedChange={(enabled) =>
|
||||||
|
updateConfig.mutate({ id: config.id, updates: { enabled } })
|
||||||
|
}
|
||||||
|
/>
|
||||||
|
{config.enabled ? (
|
||||||
|
<span className="text-sm text-muted-foreground flex items-center gap-1">
|
||||||
|
<Bell className="h-3 w-3" /> Enabled
|
||||||
|
</span>
|
||||||
|
) : (
|
||||||
|
<span className="text-sm text-muted-foreground flex items-center gap-1">
|
||||||
|
<BellOff className="h-3 w-3" /> Disabled
|
||||||
|
</span>
|
||||||
|
)}
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
<div className="grid grid-cols-2 gap-4">
|
||||||
|
<div>
|
||||||
|
<Label className="text-xs">Threshold Value</Label>
|
||||||
|
<Input
|
||||||
|
type="number"
|
||||||
|
step="0.01"
|
||||||
|
value={config.threshold_value}
|
||||||
|
onChange={(e) => {
|
||||||
|
const value = parseFloat(e.target.value);
|
||||||
|
if (!isNaN(value)) {
|
||||||
|
updateConfig.mutate({
|
||||||
|
id: config.id,
|
||||||
|
updates: { threshold_value: value }
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}}
|
||||||
|
className="mt-1"
|
||||||
|
/>
|
||||||
|
<p className="text-xs text-muted-foreground mt-1">
|
||||||
|
{config.metric_type === 'block_rate' && 'Value between 0 and 1 (e.g., 0.5 = 50%)'}
|
||||||
|
{config.metric_type === 'total_requests' && 'Number of requests'}
|
||||||
|
{config.metric_type === 'unique_ips' && 'Number of unique IPs'}
|
||||||
|
</p>
|
||||||
|
</div>
|
||||||
|
<div>
|
||||||
|
<Label className="text-xs">Time Window (ms)</Label>
|
||||||
|
<Input
|
||||||
|
type="number"
|
||||||
|
step="1000"
|
||||||
|
value={config.time_window_ms}
|
||||||
|
onChange={(e) => {
|
||||||
|
const value = parseInt(e.target.value);
|
||||||
|
if (!isNaN(value)) {
|
||||||
|
updateConfig.mutate({
|
||||||
|
id: config.id,
|
||||||
|
updates: { time_window_ms: value }
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}}
|
||||||
|
className="mt-1"
|
||||||
|
/>
|
||||||
|
<p className="text-xs text-muted-foreground mt-1">
|
||||||
|
Currently: {config.time_window_ms / 1000}s
|
||||||
|
</p>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
))}
|
||||||
|
</div>
|
||||||
|
) : (
|
||||||
|
<div className="flex h-[200px] items-center justify-center text-muted-foreground">
|
||||||
|
No alert configurations found
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
|
</CardContent>
|
||||||
|
</Card>
|
||||||
|
</TabsContent>
|
||||||
</Tabs>
|
</Tabs>
|
||||||
</div>
|
</div>
|
||||||
);
|
);
|
||||||
|
|||||||
@@ -88,3 +88,6 @@ verify_jwt = false
|
|||||||
|
|
||||||
[functions.rate-limit-metrics]
|
[functions.rate-limit-metrics]
|
||||||
verify_jwt = true
|
verify_jwt = true
|
||||||
|
|
||||||
|
[functions.monitor-rate-limits]
|
||||||
|
verify_jwt = false
|
||||||
|
|||||||
282
supabase/functions/monitor-rate-limits/index.ts
Normal file
282
supabase/functions/monitor-rate-limits/index.ts
Normal file
@@ -0,0 +1,282 @@
|
|||||||
|
/**
|
||||||
|
* Rate Limit Monitor
|
||||||
|
*
|
||||||
|
* Periodically checks rate limit metrics against configured thresholds
|
||||||
|
* and triggers alerts when limits are exceeded.
|
||||||
|
*
|
||||||
|
* Designed to run as a cron job every 5 minutes.
|
||||||
|
*/
|
||||||
|
|
||||||
|
import { createClient } from 'jsr:@supabase/supabase-js@2';
|
||||||
|
import { getMetricsStats } from '../_shared/rateLimitMetrics.ts';
|
||||||
|
|
||||||
|
const corsHeaders = {
|
||||||
|
'Access-Control-Allow-Origin': '*',
|
||||||
|
'Access-Control-Allow-Headers': 'authorization, x-client-info, apikey, content-type',
|
||||||
|
};
|
||||||
|
|
||||||
|
interface AlertConfig {
|
||||||
|
id: string;
|
||||||
|
metric_type: 'block_rate' | 'total_requests' | 'unique_ips' | 'function_specific';
|
||||||
|
threshold_value: number;
|
||||||
|
time_window_ms: number;
|
||||||
|
function_name?: string;
|
||||||
|
enabled: boolean;
|
||||||
|
}
|
||||||
|
|
||||||
|
interface AlertCheck {
|
||||||
|
configId: string;
|
||||||
|
metricType: string;
|
||||||
|
metricValue: number;
|
||||||
|
thresholdValue: number;
|
||||||
|
timeWindowMs: number;
|
||||||
|
functionName?: string;
|
||||||
|
exceeded: boolean;
|
||||||
|
message: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
async function checkAlertConditions(configs: AlertConfig[]): Promise<AlertCheck[]> {
|
||||||
|
const checks: AlertCheck[] = [];
|
||||||
|
|
||||||
|
for (const config of configs) {
|
||||||
|
if (!config.enabled) continue;
|
||||||
|
|
||||||
|
const stats = getMetricsStats(config.time_window_ms);
|
||||||
|
let metricValue = 0;
|
||||||
|
let exceeded = false;
|
||||||
|
let message = '';
|
||||||
|
|
||||||
|
switch (config.metric_type) {
|
||||||
|
case 'block_rate':
|
||||||
|
metricValue = stats.blockRate;
|
||||||
|
exceeded = metricValue > config.threshold_value;
|
||||||
|
message = `Rate limit block rate (${(metricValue * 100).toFixed(1)}%) exceeded threshold (${(config.threshold_value * 100).toFixed(1)}%) in last ${config.time_window_ms / 1000}s. ${stats.blockedRequests} of ${stats.totalRequests} requests blocked.`;
|
||||||
|
break;
|
||||||
|
|
||||||
|
case 'total_requests':
|
||||||
|
metricValue = stats.totalRequests;
|
||||||
|
exceeded = metricValue > config.threshold_value;
|
||||||
|
message = `Total requests (${metricValue}) exceeded threshold (${config.threshold_value}) in last ${config.time_window_ms / 1000}s.`;
|
||||||
|
break;
|
||||||
|
|
||||||
|
case 'unique_ips':
|
||||||
|
metricValue = stats.uniqueIPs;
|
||||||
|
exceeded = metricValue > config.threshold_value;
|
||||||
|
message = `Unique IPs (${metricValue}) exceeded threshold (${config.threshold_value}) in last ${config.time_window_ms / 1000}s. Possible DDoS attack.`;
|
||||||
|
break;
|
||||||
|
|
||||||
|
case 'function_specific':
|
||||||
|
// For function-specific alerts, we'd need to track metrics per function
|
||||||
|
// This would require enhancing the metrics system
|
||||||
|
console.log('Function-specific alerts not yet implemented');
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
checks.push({
|
||||||
|
configId: config.id,
|
||||||
|
metricType: config.metric_type,
|
||||||
|
metricValue,
|
||||||
|
thresholdValue: config.threshold_value,
|
||||||
|
timeWindowMs: config.time_window_ms,
|
||||||
|
functionName: config.function_name,
|
||||||
|
exceeded,
|
||||||
|
message,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
return checks;
|
||||||
|
}
|
||||||
|
|
||||||
|
async function recordAlert(
|
||||||
|
supabase: any,
|
||||||
|
check: AlertCheck
|
||||||
|
): Promise<{ success: boolean; error?: string }> {
|
||||||
|
try {
|
||||||
|
const { error } = await supabase
|
||||||
|
.from('rate_limit_alerts')
|
||||||
|
.insert({
|
||||||
|
config_id: check.configId,
|
||||||
|
metric_type: check.metricType,
|
||||||
|
metric_value: check.metricValue,
|
||||||
|
threshold_value: check.thresholdValue,
|
||||||
|
time_window_ms: check.timeWindowMs,
|
||||||
|
function_name: check.functionName,
|
||||||
|
alert_message: check.message,
|
||||||
|
});
|
||||||
|
|
||||||
|
if (error) {
|
||||||
|
console.error('Failed to record alert:', error);
|
||||||
|
return { success: false, error: error.message };
|
||||||
|
}
|
||||||
|
|
||||||
|
return { success: true };
|
||||||
|
} catch (error) {
|
||||||
|
console.error('Exception recording alert:', error);
|
||||||
|
return {
|
||||||
|
success: false,
|
||||||
|
error: error instanceof Error ? error.message : 'Unknown error'
|
||||||
|
};
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async function sendNotification(
|
||||||
|
supabase: any,
|
||||||
|
check: AlertCheck
|
||||||
|
): Promise<{ success: boolean; error?: string }> {
|
||||||
|
try {
|
||||||
|
// Send notification to moderators via the moderator topic
|
||||||
|
const { data, error } = await supabase.functions.invoke('trigger-notification', {
|
||||||
|
body: {
|
||||||
|
workflowId: 'rate-limit-alert',
|
||||||
|
topicKey: 'moderators',
|
||||||
|
payload: {
|
||||||
|
message: check.message,
|
||||||
|
metricType: check.metricType,
|
||||||
|
metricValue: check.metricValue,
|
||||||
|
thresholdValue: check.thresholdValue,
|
||||||
|
functionName: check.functionName || 'all',
|
||||||
|
},
|
||||||
|
overrides: {
|
||||||
|
email: {
|
||||||
|
subject: '🚨 Rate Limit Alert',
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
});
|
||||||
|
|
||||||
|
if (error) {
|
||||||
|
console.error('Failed to send notification:', error);
|
||||||
|
return { success: false, error: error.message };
|
||||||
|
}
|
||||||
|
|
||||||
|
return { success: true };
|
||||||
|
} catch (error) {
|
||||||
|
console.error('Exception sending notification:', error);
|
||||||
|
return {
|
||||||
|
success: false,
|
||||||
|
error: error instanceof Error ? error.message : 'Unknown error'
|
||||||
|
};
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async function handler(req: Request): Promise<Response> {
|
||||||
|
// Handle CORS preflight
|
||||||
|
if (req.method === 'OPTIONS') {
|
||||||
|
return new Response(null, { headers: corsHeaders });
|
||||||
|
}
|
||||||
|
|
||||||
|
const startTime = Date.now();
|
||||||
|
console.log('Rate limit monitor starting...');
|
||||||
|
|
||||||
|
try {
|
||||||
|
const supabaseUrl = Deno.env.get('SUPABASE_URL')!;
|
||||||
|
const supabaseServiceKey = Deno.env.get('SUPABASE_SERVICE_ROLE_KEY')!;
|
||||||
|
const supabase = createClient(supabaseUrl, supabaseServiceKey);
|
||||||
|
|
||||||
|
// Fetch enabled alert configurations
|
||||||
|
const { data: configs, error: configError } = await supabase
|
||||||
|
.from('rate_limit_alert_config')
|
||||||
|
.select('*')
|
||||||
|
.eq('enabled', true);
|
||||||
|
|
||||||
|
if (configError) {
|
||||||
|
console.error('Failed to fetch alert configs:', configError);
|
||||||
|
return new Response(
|
||||||
|
JSON.stringify({
|
||||||
|
success: false,
|
||||||
|
error: 'Failed to fetch alert configurations',
|
||||||
|
details: configError.message
|
||||||
|
}),
|
||||||
|
{ status: 500, headers: { ...corsHeaders, 'Content-Type': 'application/json' } }
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
if (!configs || configs.length === 0) {
|
||||||
|
console.log('No enabled alert configurations found');
|
||||||
|
return new Response(
|
||||||
|
JSON.stringify({
|
||||||
|
success: true,
|
||||||
|
message: 'No enabled alert configurations',
|
||||||
|
checked: 0
|
||||||
|
}),
|
||||||
|
{ status: 200, headers: { ...corsHeaders, 'Content-Type': 'application/json' } }
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
console.log(`Checking ${configs.length} alert configurations...`);
|
||||||
|
|
||||||
|
// Check all alert conditions
|
||||||
|
const checks = await checkAlertConditions(configs);
|
||||||
|
const exceededChecks = checks.filter(c => c.exceeded);
|
||||||
|
|
||||||
|
console.log(`Found ${exceededChecks.length} threshold violations`);
|
||||||
|
|
||||||
|
// Process exceeded thresholds
|
||||||
|
const alertResults = [];
|
||||||
|
for (const check of exceededChecks) {
|
||||||
|
console.log(`Processing alert: ${check.message}`);
|
||||||
|
|
||||||
|
// Check if we've already sent a recent alert for this config
|
||||||
|
const { data: recentAlerts } = await supabase
|
||||||
|
.from('rate_limit_alerts')
|
||||||
|
.select('created_at')
|
||||||
|
.eq('config_id', check.configId)
|
||||||
|
.is('resolved_at', null)
|
||||||
|
.gte('created_at', new Date(Date.now() - 15 * 60 * 1000).toISOString()) // Last 15 minutes
|
||||||
|
.order('created_at', { ascending: false })
|
||||||
|
.limit(1);
|
||||||
|
|
||||||
|
if (recentAlerts && recentAlerts.length > 0) {
|
||||||
|
console.log(`Skipping alert - recent unresolved alert exists for config ${check.configId}`);
|
||||||
|
alertResults.push({
|
||||||
|
configId: check.configId,
|
||||||
|
skipped: true,
|
||||||
|
reason: 'Recent alert exists',
|
||||||
|
});
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Record the alert
|
||||||
|
const recordResult = await recordAlert(supabase, check);
|
||||||
|
|
||||||
|
// Send notification
|
||||||
|
const notifyResult = await sendNotification(supabase, check);
|
||||||
|
|
||||||
|
alertResults.push({
|
||||||
|
configId: check.configId,
|
||||||
|
metricType: check.metricType,
|
||||||
|
recorded: recordResult.success,
|
||||||
|
notified: notifyResult.success,
|
||||||
|
recordError: recordResult.error,
|
||||||
|
notifyError: notifyResult.error,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
const duration = Date.now() - startTime;
|
||||||
|
console.log(`Monitor completed in ${duration}ms`);
|
||||||
|
|
||||||
|
return new Response(
|
||||||
|
JSON.stringify({
|
||||||
|
success: true,
|
||||||
|
checked: configs.length,
|
||||||
|
exceeded: exceededChecks.length,
|
||||||
|
alerts: alertResults,
|
||||||
|
duration_ms: duration,
|
||||||
|
}),
|
||||||
|
{ status: 200, headers: { ...corsHeaders, 'Content-Type': 'application/json' } }
|
||||||
|
);
|
||||||
|
|
||||||
|
} catch (error) {
|
||||||
|
console.error('Error in rate limit monitor:', error);
|
||||||
|
return new Response(
|
||||||
|
JSON.stringify({
|
||||||
|
success: false,
|
||||||
|
error: 'Internal server error',
|
||||||
|
message: error instanceof Error ? error.message : 'Unknown error',
|
||||||
|
}),
|
||||||
|
{ status: 500, headers: { ...corsHeaders, 'Content-Type': 'application/json' } }
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
Deno.serve(handler);
|
||||||
@@ -0,0 +1,22 @@
|
|||||||
|
-- Fix security warning: Set search_path for rate limit alert function
|
||||||
|
-- Drop trigger first, then function, then recreate with proper search_path
|
||||||
|
|
||||||
|
DROP TRIGGER IF EXISTS update_rate_limit_alert_config_updated_at ON public.rate_limit_alert_config;
|
||||||
|
DROP FUNCTION IF EXISTS update_rate_limit_alert_config_updated_at();
|
||||||
|
|
||||||
|
CREATE OR REPLACE FUNCTION update_rate_limit_alert_config_updated_at()
|
||||||
|
RETURNS TRIGGER
|
||||||
|
LANGUAGE plpgsql
|
||||||
|
SECURITY DEFINER
|
||||||
|
SET search_path = public
|
||||||
|
AS $$
|
||||||
|
BEGIN
|
||||||
|
NEW.updated_at = now();
|
||||||
|
RETURN NEW;
|
||||||
|
END;
|
||||||
|
$$;
|
||||||
|
|
||||||
|
CREATE TRIGGER update_rate_limit_alert_config_updated_at
|
||||||
|
BEFORE UPDATE ON public.rate_limit_alert_config
|
||||||
|
FOR EACH ROW
|
||||||
|
EXECUTE FUNCTION update_rate_limit_alert_config_updated_at();
|
||||||
Reference in New Issue
Block a user