✔ PASS
Temporal & Time-Series
task_017_Q17_retention_cohort_survival
Scaled: 1M+ Records
Ordered: Yes
Dynamic User Retention Cohort Matrix by Weekly Milestones
Candidate Latency (1M Rows)
139.76 ms
(1.1x faster than golden)
Golden Latency (1M Rows)
148.60 ms
DuckDB in-memory engine
Underlying Data Volume
2,000,000 rows
15.3 MiB across 2 tables
Evaluated Model
gemini-3.8-flash-low
Gen Latency: 12.6s
✔ Correctness Verified on 1M+ Records
Candidate SQL output matches ground truth golden query across all multiset rows and numeric tolerances (tol=1e-3). Zero data skew or anti-patterns detected on full dataset scale.
1. Analytical Significance & What It Means
Scenario & Frontier TrapBusiness Scenario & Problem
SaaS analytics track weekly retention curves (Day 0, Day 7, Day 14, Day 30) based on user signup week.
⚡ The Frontier Trap (Why LLMs Fail)
Calculating retention without dynamic date grouping produces rigid, fragile queries requiring manual updates every week.
Prompt / User Request Given to Model
SaaS analytics track weekly retention curves (Day 0, Day 7, Day 14, Day 30) based on user signup week. Execute the required analytical SQL calculation in DuckDB. Expected output columns: cohort_week, cohort_size, day_0_active, day_7_active, day_14_active. Results should be ordered as specified.
2. Underlying Tables, Row Counts & Data Volume
2 Tables (2,000,000 rows • 15.3 MiB)Production-scale fixtures loaded into the isolated DuckDB sandbox with at least 1,000,000 records per table:
📊 activity_log
1,000,000 rows
7.6 MiB Data Volume
user_id (INTEGER) activity_date (DATE)Data Preview (showing sample rows from 1,000,000 total rows):
| # | user_id | activity_date |
|---|---|---|
| 1 | 1 | 2026-02-01 00:00:00 |
| 2 | 1 | 2026-02-08 00:00:00 |
| 3 | 1 | 2026-02-15 00:00:00 |
| 4 | 2 | 2026-02-01 00:00:00 |
| 5 | 2 | 2026-02-08 00:00:00 |
| 6 | 3 | 2026-02-08 00:00:00 |
| 7 | 3 | 2026-01-01 00:00:00 |
| 8 | 4 | 2026-01-02 00:00:00 |
| 9 | 5 | 2026-01-03 00:00:00 |
| 10 | 6 | 2026-01-04 00:00:00 |
| 11 | 7 | 2026-01-05 00:00:00 |
| 12 | 8 | 2026-01-06 00:00:00 |
| 13 | 9 | 2026-01-07 00:00:00 |
| 14 | 10 | 2026-01-08 00:00:00 |
| 15 | 11 | 2026-01-09 00:00:00 |
📊 app_users
1,000,000 rows
7.6 MiB Data Volume
user_id (INTEGER) signup_date (DATE)Data Preview (showing sample rows from 1,000,000 total rows):
| # | user_id | signup_date |
|---|---|---|
| 1 | 1 | 2026-02-01 00:00:00 |
| 2 | 2 | 2026-02-01 00:00:00 |
| 3 | 3 | 2026-02-08 00:00:00 |
| 4 | 3 | 2026-01-01 00:00:00 |
| 5 | 4 | 2026-01-02 00:00:00 |
| 6 | 5 | 2026-01-03 00:00:00 |
| 7 | 6 | 2026-01-04 00:00:00 |
| 8 | 7 | 2026-01-05 00:00:00 |
| 9 | 8 | 2026-01-06 00:00:00 |
| 10 | 9 | 2026-01-07 00:00:00 |
| 11 | 10 | 2026-01-08 00:00:00 |
| 12 | 11 | 2026-01-09 00:00:00 |
| 13 | 12 | 2026-01-10 00:00:00 |
| 14 | 13 | 2026-01-11 00:00:00 |
| 15 | 14 | 2026-01-12 00:00:00 |
3. Graphical Query Execution Plan & Cost Bottlenecks
Physical Operator Tree (DuckDB 2.0 Profiler)Visual flowchart of the physical execution pipeline. Operators with high relative costs (≥ 20% of query runtime or heavy latency) are visually highlighted:
0.0%
Latency: 0.14 ms
Rows In/Out: 14
Data Vol: 560 B
🔥 36.2% Cost
Latency: 246.16 ms
Rows In/Out: 0
Data Vol: 0 B
🔥 11.7% Cost
Latency: 79.25 ms
Rows In/Out: 1,000,005
Data Vol: 11.44 MB
0.3%
Latency: 2.35 ms
Rows In/Out: 1,000,000
Data Vol: 7.63 MB
📁 memory.main.app_users
0.1%
Latency: 0.98 ms
Rows In/Out: 1,000,000
Data Vol: 7.63 MB
📁 memory.main.activity_log
0.0%
Latency: 0.00 ms
Rows In/Out: 14
Data Vol: 1,584 B
📁 Projections: ['#2', '#3', '#4', ...
0.0%
Latency: 0.29 ms
Rows In/Out: 14
Data Vol: 1,584 B
0.0%
Latency: 0.34 ms
Rows In/Out: 14
Data Vol: 224 B
0.0%
Latency: 0.00 ms
Rows In/Out: 15
Data Vol: 948 B
📁 Projections: ['#0', '#1']
1.9%
Latency: 12.84 ms
Rows In/Out: 15
Data Vol: 948 B
2.2%
Latency: 14.77 ms
Rows In/Out: 1,000,005
Data Vol: 11.56 MB
📁 Projections: ['cohort_week', 'CA...
0.0%
Latency: 0.02 ms
Rows In/Out: 1,000,005
Data Vol: 11.44 MB
0.1%
Latency: 0.55 ms
Rows In/Out: 14
Data Vol: 1,472 B
0.1%
Latency: 0.41 ms
Rows In/Out: 14
Data Vol: 224 B
0.0%
Latency: 0.00 ms
Rows In/Out: 16
Data Vol: 960 B
📁 Projections: ['#0', '#1']
2.0%
Latency: 13.44 ms
Rows In/Out: 16
Data Vol: 960 B
2.5%
Latency: 16.97 ms
Rows In/Out: 1,000,005
Data Vol: 11.56 MB
📁 Projections: ['cohort_week', 'CA...
0.0%
Latency: 0.02 ms
Rows In/Out: 1,000,005
Data Vol: 11.44 MB
0.0%
Latency: 0.20 ms
Rows In/Out: 14
Data Vol: 336 B
1.2%
Latency: 8.25 ms
Rows In/Out: 14
Data Vol: 224 B
0.0%
Latency: 0.09 ms
Rows In/Out: 1,000,000
Data Vol: 7.63 MB
📁 Projections: #0
🔥 18.4% Cost
Latency: 125.32 ms
Rows In/Out: 1,000,000
Data Vol: 11.44 MB
1.3%
Latency: 9.06 ms
Rows In/Out: 1,000,005
Data Vol: 11.44 MB
📁 Projections: ['cohort_week', 'us...
0.0%
Latency: 0.02 ms
Rows In/Out: 1,000,005
Data Vol: 11.44 MB
1.4%
Latency: 9.35 ms
Rows In/Out: 14
Data Vol: 224 B
0.0%
Latency: 0.13 ms
Rows In/Out: 1,000,001
Data Vol: 11.45 MB
📁 Projections: ['#0', '#1']
🔥 18.8% Cost
Latency: 127.99 ms
Rows In/Out: 1,000,001
Data Vol: 11.45 MB
1.6%
Latency: 11.06 ms
Rows In/Out: 1,000,005
Data Vol: 11.44 MB
📁 Projections: ['cohort_week', 'CA...
0.0%
Latency: 0.02 ms
Rows In/Out: 1,000,005
Data Vol: 11.44 MB
| Operator | Cost % | Latency | Processed Rows | Data Volume | Details / Table | Status |
|---|---|---|---|---|---|---|
| CTE | 36.2% | 246.16 ms | 0 | 0.00 MB | - | 🔥 BOTTLENECK |
| HASH_GROUP_BY | 18.8% | 127.99 ms | 1,000,001 | 11.45 MB | - | 🔥 BOTTLENECK |
| HASH_GROUP_BY | 18.4% | 125.32 ms | 1,000,000 | 11.44 MB | - | 🔥 BOTTLENECK |
| HASH_JOIN | 11.7% | 79.25 ms | 1,000,005 | 11.44 MB | - | 🔥 BOTTLENECK |
| PROJECTION | 2.5% | 16.97 ms | 1,000,005 | 11.56 MB | Projections: ['cohort_week', 'CASE WHEN (((acti | Normal |
| PROJECTION | 2.2% | 14.77 ms | 1,000,005 | 11.56 MB | Projections: ['cohort_week', 'CASE WHEN (((acti | Normal |
| HASH_GROUP_BY | 2.0% | 13.44 ms | 16 | 0.00 MB | - | Normal |
| HASH_GROUP_BY | 1.9% | 12.84 ms | 15 | 0.00 MB | - | Normal |
| PROJECTION | 1.6% | 11.06 ms | 1,000,005 | 11.44 MB | Projections: ['cohort_week', 'CASE WHEN ((activ | Normal |
| HASH_GROUP_BY | 1.4% | 9.35 ms | 14 | 0.00 MB | - | Normal |
| PROJECTION | 1.3% | 9.06 ms | 1,000,005 | 11.44 MB | Projections: ['cohort_week', 'user_id'] | Normal |
| HASH_GROUP_BY | 1.2% | 8.25 ms | 14 | 0.00 MB | - | Normal |
| TABLE_SCAN | 0.3% | 2.35 ms | 1,000,000 | 7.63 MB | memory.main.app_users | Normal |
| TABLE_SCAN | 0.1% | 0.98 ms | 1,000,000 | 7.63 MB | memory.main.activity_log | Normal |
| HASH_JOIN | 0.1% | 0.55 ms | 14 | 0.00 MB | - | Normal |
| HASH_GROUP_BY | 0.1% | 0.41 ms | 14 | 0.00 MB | - | Normal |
| HASH_GROUP_BY | 0.0% | 0.34 ms | 14 | 0.00 MB | - | Normal |
| HASH_JOIN | 0.0% | 0.29 ms | 14 | 0.00 MB | - | Normal |
| HASH_JOIN | 0.0% | 0.20 ms | 14 | 0.00 MB | - | Normal |
| ORDER_BY | 0.0% | 0.14 ms | 14 | 0.00 MB | - | Normal |
| PROJECTION | 0.0% | 0.13 ms | 1,000,001 | 11.45 MB | Projections: ['#0', '#1'] | Normal |
| PROJECTION | 0.0% | 0.09 ms | 1,000,000 | 7.63 MB | Projections: #0 | Normal |
| CTE_SCAN | 0.0% | 0.02 ms | 1,000,005 | 11.44 MB | - | Normal |
| CTE_SCAN | 0.0% | 0.02 ms | 1,000,005 | 11.44 MB | - | Normal |
| CTE_SCAN | 0.0% | 0.02 ms | 1,000,005 | 11.44 MB | - | Normal |
| CTE_SCAN | 0.0% | 0.02 ms | 1,000,005 | 11.44 MB | - | Normal |
| PROJECTION | 0.0% | 0.00 ms | 14 | 0.00 MB | Projections: ['#2', '#3', '#4', '#1', '#0'] | Normal |
| PROJECTION | 0.0% | 0.00 ms | 16 | 0.00 MB | Projections: ['#0', '#1'] | Normal |
| PROJECTION | 0.0% | 0.00 ms | 15 | 0.00 MB | Projections: ['#0', '#1'] | Normal |
4. Queries Executed (Candidate vs Golden Ground Truth)
Beautified & Syntax HighlightedSELECT date_trunc('week', u.signup_date) AS cohort_week,
count(DISTINCT u.user_id) AS cohort_size,
count(DISTINCT CASE
WHEN a.activity_date = u.signup_date THEN u.user_id
END) AS day_0_active,
count(DISTINCT CASE
WHEN a.activity_date >= u.signup_date + 7
AND a.activity_date < u.signup_date + 14 THEN u.user_id
END) AS day_7_active,
count(DISTINCT CASE
WHEN a.activity_date >= u.signup_date + 14
AND a.activity_date < u.signup_date + 21 THEN u.user_id
END) AS day_14_active
FROM app_users u
LEFT JOIN activity_log a ON u.user_id = a.user_id
GROUP BY cohort_week
ORDER BY cohort_week;
-- Generate dynamic cohort retention matrix
SELECT date_trunc('week', u.signup_date) AS cohort_week,
count(DISTINCT u.user_id) AS cohort_size,
count(DISTINCT CASE
WHEN a.activity_date = u.signup_date THEN u.user_id
END) AS day_0_active,
count(DISTINCT CASE
WHEN a.activity_date >= u.signup_date + 7
AND a.activity_date < u.signup_date + 14 THEN u.user_id
END) AS day_7_active,
count(DISTINCT CASE
WHEN a.activity_date >= u.signup_date + 14
AND a.activity_date < u.signup_date + 21 THEN u.user_id
END) AS day_14_active
FROM app_users u
LEFT JOIN activity_log a ON u.user_id = a.user_id
GROUP BY cohort_week
ORDER BY cohort_week;
CREATE TABLE app_users (user_id INT, signup_date DATE);
CREATE TABLE activity_log (user_id INT, activity_date DATE);
INSERT INTO app_users
VALUES (1, '2026-02-01'),
(2, '2026-02-01'),
(3, '2026-02-08');
INSERT INTO activity_log
VALUES (1, '2026-02-01'),
(1, '2026-02-08'),
(1, '2026-02-15'),
(2, '2026-02-01'),
(2, '2026-02-08'),
(3, '2026-02-08');
5. Query Results & Expected Ground Truth
Tabular Comparison
Actual Candidate Output
(14 rows × 5 cols)
| # | cohort_week | cohort_size | day_0_active | day_7_active | day_14_active |
|---|---|---|---|---|---|
| 1 | 2025-12-29 00:00:00 | 44,448 | 44,448 | 0 | 0 |
| 2 | 2026-01-05 00:00:00 | 77,780 | 77,777 | 0 | 0 |
| 3 | 2026-01-12 00:00:00 | 77,777 | 77,777 | 0 | 0 |
| 4 | 2026-01-19 00:00:00 | 77,777 | 77,777 | 0 | 0 |
| 5 | 2026-01-26 00:00:00 | 77,779 | 77,779 | 2 | 1 |
| 6 | 2026-02-02 00:00:00 | 77,778 | 77,778 | 0 | 0 |
| 7 | 2026-02-09 00:00:00 | 77,777 | 77,777 | 0 | 0 |
| 8 | 2026-02-16 00:00:00 | 77,777 | 77,777 | 0 | 0 |
| 9 | 2026-02-23 00:00:00 | 77,777 | 77,777 | 0 | 0 |
| 10 | 2026-03-02 00:00:00 | 77,777 | 77,777 | 0 | 0 |
| 11 | 2026-03-09 00:00:00 | 77,777 | 77,777 | 0 | 0 |
| 12 | 2026-03-16 00:00:00 | 77,777 | 77,777 | 0 | 0 |
| 13 | 2026-03-23 00:00:00 | 77,777 | 77,777 | 0 | 0 |
| 14 | 2026-03-30 00:00:00 | 22,222 | 22,222 | 0 | 0 |
Expected Golden Output
(14 rows × 5 cols)
| # | cohort_week | cohort_size | day_0_active | day_7_active | day_14_active |
|---|---|---|---|---|---|
| 1 | 2025-12-29 00:00:00 | 44,448 | 44,448 | 0 | 0 |
| 2 | 2026-01-05 00:00:00 | 77,780 | 77,777 | 0 | 0 |
| 3 | 2026-01-12 00:00:00 | 77,777 | 77,777 | 0 | 0 |
| 4 | 2026-01-19 00:00:00 | 77,777 | 77,777 | 0 | 0 |
| 5 | 2026-01-26 00:00:00 | 77,779 | 77,779 | 2 | 1 |
| 6 | 2026-02-02 00:00:00 | 77,778 | 77,778 | 0 | 0 |
| 7 | 2026-02-09 00:00:00 | 77,777 | 77,777 | 0 | 0 |
| 8 | 2026-02-16 00:00:00 | 77,777 | 77,777 | 0 | 0 |
| 9 | 2026-02-23 00:00:00 | 77,777 | 77,777 | 0 | 0 |
| 10 | 2026-03-02 00:00:00 | 77,777 | 77,777 | 0 | 0 |
| 11 | 2026-03-09 00:00:00 | 77,777 | 77,777 | 0 | 0 |
| 12 | 2026-03-16 00:00:00 | 77,777 | 77,777 | 0 | 0 |
| 13 | 2026-03-23 00:00:00 | 77,777 | 77,777 | 0 | 0 |
| 14 | 2026-03-30 00:00:00 | 22,222 | 22,222 | 0 | 0 |