✔ PASS Temporal & Time-Series task_017_Q17_retention_cohort_survival Scaled: 1M+ Records Ordered: Yes

Dynamic User Retention Cohort Matrix by Weekly Milestones

Candidate Latency (1M Rows)
139.76 ms
(1.1x faster than golden)
Golden Latency (1M Rows)
148.60 ms
DuckDB in-memory engine
Underlying Data Volume
2,000,000 rows
15.3 MiB across 2 tables
Evaluated Model
gemini-3.8-flash-low
Gen Latency: 12.6s
✔ Correctness Verified on 1M+ Records
Candidate SQL output matches ground truth golden query across all multiset rows and numeric tolerances (tol=1e-3). Zero data skew or anti-patterns detected on full dataset scale.

1. Analytical Significance & What It Means

Scenario & Frontier Trap
Business Scenario & Problem

SaaS analytics track weekly retention curves (Day 0, Day 7, Day 14, Day 30) based on user signup week.

⚡ The Frontier Trap (Why LLMs Fail)

Calculating retention without dynamic date grouping produces rigid, fragile queries requiring manual updates every week.

Prompt / User Request Given to Model
SaaS analytics track weekly retention curves (Day 0, Day 7, Day 14, Day 30) based on user signup week.

Execute the required analytical SQL calculation in DuckDB.
Expected output columns: cohort_week, cohort_size, day_0_active, day_7_active, day_14_active.
Results should be ordered as specified.

2. Underlying Tables, Row Counts & Data Volume

2 Tables (2,000,000 rows • 15.3 MiB)

Production-scale fixtures loaded into the isolated DuckDB sandbox with at least 1,000,000 records per table:

📊 activity_log 1,000,000 rows 7.6 MiB Data Volume
user_id (INTEGER) activity_date (DATE)
Data Preview (showing sample rows from 1,000,000 total rows):
# user_id activity_date
1 1 2026-02-01 00:00:00
2 1 2026-02-08 00:00:00
3 1 2026-02-15 00:00:00
4 2 2026-02-01 00:00:00
5 2 2026-02-08 00:00:00
6 3 2026-02-08 00:00:00
7 3 2026-01-01 00:00:00
8 4 2026-01-02 00:00:00
9 5 2026-01-03 00:00:00
10 6 2026-01-04 00:00:00
11 7 2026-01-05 00:00:00
12 8 2026-01-06 00:00:00
13 9 2026-01-07 00:00:00
14 10 2026-01-08 00:00:00
15 11 2026-01-09 00:00:00
📊 app_users 1,000,000 rows 7.6 MiB Data Volume
user_id (INTEGER) signup_date (DATE)
Data Preview (showing sample rows from 1,000,000 total rows):
# user_id signup_date
1 1 2026-02-01 00:00:00
2 2 2026-02-01 00:00:00
3 3 2026-02-08 00:00:00
4 3 2026-01-01 00:00:00
5 4 2026-01-02 00:00:00
6 5 2026-01-03 00:00:00
7 6 2026-01-04 00:00:00
8 7 2026-01-05 00:00:00
9 8 2026-01-06 00:00:00
10 9 2026-01-07 00:00:00
11 10 2026-01-08 00:00:00
12 11 2026-01-09 00:00:00
13 12 2026-01-10 00:00:00
14 13 2026-01-11 00:00:00
15 14 2026-01-12 00:00:00

3. Graphical Query Execution Plan & Cost Bottlenecks

Physical Operator Tree (DuckDB 2.0 Profiler)

Visual flowchart of the physical execution pipeline. Operators with high relative costs (≥ 20% of query runtime or heavy latency) are visually highlighted:

Pipeline Latency Distribution (Total Plan Time: 680.02 ms) Nodes consuming ≥ 20% of query runtime are highlighted with glowing borders
⏳ ORDER_BY 0.0%
Latency: 0.14 ms
Rows In/Out: 14
Data Vol: 560 B
🔄 CTE 🔥 36.2% Cost
Latency: 246.16 ms
Rows In/Out: 0
Data Vol: 0 B
⚠️ Heavy Execution Bottleneck (36.2% of query runtime)
⚡ HASH_JOIN 🔥 11.7% Cost
Latency: 79.25 ms
Rows In/Out: 1,000,005
Data Vol: 11.44 MB
⚠️ Heavy Execution Bottleneck (11.7% of query runtime)
🔍 TABLE_SCAN 0.3%
Latency: 2.35 ms
Rows In/Out: 1,000,000
Data Vol: 7.63 MB
📁 memory.main.app_users
🔍 TABLE_SCAN 0.1%
Latency: 0.98 ms
Rows In/Out: 1,000,000
Data Vol: 7.63 MB
📁 memory.main.activity_log
📐 PROJECTION 0.0%
Latency: 0.00 ms
Rows In/Out: 14
Data Vol: 1,584 B
📁 Projections: ['#2', '#3', '#4', ...
⚡ HASH_JOIN 0.0%
Latency: 0.29 ms
Rows In/Out: 14
Data Vol: 1,584 B
📊 HASH_GROUP_BY 0.0%
Latency: 0.34 ms
Rows In/Out: 14
Data Vol: 224 B
📐 PROJECTION 0.0%
Latency: 0.00 ms
Rows In/Out: 15
Data Vol: 948 B
📁 Projections: ['#0', '#1']
📊 HASH_GROUP_BY 1.9%
Latency: 12.84 ms
Rows In/Out: 15
Data Vol: 948 B
📐 PROJECTION 2.2%
Latency: 14.77 ms
Rows In/Out: 1,000,005
Data Vol: 11.56 MB
📁 Projections: ['cohort_week', 'CA...
🔍 CTE_SCAN 0.0%
Latency: 0.02 ms
Rows In/Out: 1,000,005
Data Vol: 11.44 MB
⚡ HASH_JOIN 0.1%
Latency: 0.55 ms
Rows In/Out: 14
Data Vol: 1,472 B
📊 HASH_GROUP_BY 0.1%
Latency: 0.41 ms
Rows In/Out: 14
Data Vol: 224 B
📐 PROJECTION 0.0%
Latency: 0.00 ms
Rows In/Out: 16
Data Vol: 960 B
📁 Projections: ['#0', '#1']
📊 HASH_GROUP_BY 2.0%
Latency: 13.44 ms
Rows In/Out: 16
Data Vol: 960 B
📐 PROJECTION 2.5%
Latency: 16.97 ms
Rows In/Out: 1,000,005
Data Vol: 11.56 MB
📁 Projections: ['cohort_week', 'CA...
🔍 CTE_SCAN 0.0%
Latency: 0.02 ms
Rows In/Out: 1,000,005
Data Vol: 11.44 MB
⚡ HASH_JOIN 0.0%
Latency: 0.20 ms
Rows In/Out: 14
Data Vol: 336 B
📊 HASH_GROUP_BY 1.2%
Latency: 8.25 ms
Rows In/Out: 14
Data Vol: 224 B
📐 PROJECTION 0.0%
Latency: 0.09 ms
Rows In/Out: 1,000,000
Data Vol: 7.63 MB
📁 Projections: #0
📊 HASH_GROUP_BY 🔥 18.4% Cost
Latency: 125.32 ms
Rows In/Out: 1,000,000
Data Vol: 11.44 MB
⚠️ Heavy Execution Bottleneck (18.4% of query runtime)
📐 PROJECTION 1.3%
Latency: 9.06 ms
Rows In/Out: 1,000,005
Data Vol: 11.44 MB
📁 Projections: ['cohort_week', 'us...
🔍 CTE_SCAN 0.0%
Latency: 0.02 ms
Rows In/Out: 1,000,005
Data Vol: 11.44 MB
📊 HASH_GROUP_BY 1.4%
Latency: 9.35 ms
Rows In/Out: 14
Data Vol: 224 B
📐 PROJECTION 0.0%
Latency: 0.13 ms
Rows In/Out: 1,000,001
Data Vol: 11.45 MB
📁 Projections: ['#0', '#1']
📊 HASH_GROUP_BY 🔥 18.8% Cost
Latency: 127.99 ms
Rows In/Out: 1,000,001
Data Vol: 11.45 MB
⚠️ Heavy Execution Bottleneck (18.8% of query runtime)
📐 PROJECTION 1.6%
Latency: 11.06 ms
Rows In/Out: 1,000,005
Data Vol: 11.44 MB
📁 Projections: ['cohort_week', 'CA...
🔍 CTE_SCAN 0.0%
Latency: 0.02 ms
Rows In/Out: 1,000,005
Data Vol: 11.44 MB
Operator Cost % Latency Processed Rows Data Volume Details / Table Status
CTE 36.2% 246.16 ms 0 0.00 MB - 🔥 BOTTLENECK
HASH_GROUP_BY 18.8% 127.99 ms 1,000,001 11.45 MB - 🔥 BOTTLENECK
HASH_GROUP_BY 18.4% 125.32 ms 1,000,000 11.44 MB - 🔥 BOTTLENECK
HASH_JOIN 11.7% 79.25 ms 1,000,005 11.44 MB - 🔥 BOTTLENECK
PROJECTION 2.5% 16.97 ms 1,000,005 11.56 MB Projections: ['cohort_week', 'CASE WHEN (((acti Normal
PROJECTION 2.2% 14.77 ms 1,000,005 11.56 MB Projections: ['cohort_week', 'CASE WHEN (((acti Normal
HASH_GROUP_BY 2.0% 13.44 ms 16 0.00 MB - Normal
HASH_GROUP_BY 1.9% 12.84 ms 15 0.00 MB - Normal
PROJECTION 1.6% 11.06 ms 1,000,005 11.44 MB Projections: ['cohort_week', 'CASE WHEN ((activ Normal
HASH_GROUP_BY 1.4% 9.35 ms 14 0.00 MB - Normal
PROJECTION 1.3% 9.06 ms 1,000,005 11.44 MB Projections: ['cohort_week', 'user_id'] Normal
HASH_GROUP_BY 1.2% 8.25 ms 14 0.00 MB - Normal
TABLE_SCAN 0.3% 2.35 ms 1,000,000 7.63 MB memory.main.app_users Normal
TABLE_SCAN 0.1% 0.98 ms 1,000,000 7.63 MB memory.main.activity_log Normal
HASH_JOIN 0.1% 0.55 ms 14 0.00 MB - Normal
HASH_GROUP_BY 0.1% 0.41 ms 14 0.00 MB - Normal
HASH_GROUP_BY 0.0% 0.34 ms 14 0.00 MB - Normal
HASH_JOIN 0.0% 0.29 ms 14 0.00 MB - Normal
HASH_JOIN 0.0% 0.20 ms 14 0.00 MB - Normal
ORDER_BY 0.0% 0.14 ms 14 0.00 MB - Normal
PROJECTION 0.0% 0.13 ms 1,000,001 11.45 MB Projections: ['#0', '#1'] Normal
PROJECTION 0.0% 0.09 ms 1,000,000 7.63 MB Projections: #0 Normal
CTE_SCAN 0.0% 0.02 ms 1,000,005 11.44 MB - Normal
CTE_SCAN 0.0% 0.02 ms 1,000,005 11.44 MB - Normal
CTE_SCAN 0.0% 0.02 ms 1,000,005 11.44 MB - Normal
CTE_SCAN 0.0% 0.02 ms 1,000,005 11.44 MB - Normal
PROJECTION 0.0% 0.00 ms 14 0.00 MB Projections: ['#2', '#3', '#4', '#1', '#0'] Normal
PROJECTION 0.0% 0.00 ms 16 0.00 MB Projections: ['#0', '#1'] Normal
PROJECTION 0.0% 0.00 ms 15 0.00 MB Projections: ['#0', '#1'] Normal

4. Queries Executed (Candidate vs Golden Ground Truth)

Beautified & Syntax Highlighted
Candidate SQL Executed by gemini-3.8-flash-low (Runtime on 1M Rows: 139.76 ms)
SELECT date_trunc('week', u.signup_date) AS cohort_week,
       count(DISTINCT u.user_id) AS cohort_size,
       count(DISTINCT CASE
                          WHEN a.activity_date = u.signup_date THEN u.user_id
                      END) AS day_0_active,
       count(DISTINCT CASE
                          WHEN a.activity_date >= u.signup_date + 7
                               AND a.activity_date < u.signup_date + 14 THEN u.user_id
                      END) AS day_7_active,
       count(DISTINCT CASE
                          WHEN a.activity_date >= u.signup_date + 14
                               AND a.activity_date < u.signup_date + 21 THEN u.user_id
                      END) AS day_14_active
FROM app_users u
LEFT JOIN activity_log a ON u.user_id = a.user_id
GROUP BY cohort_week
ORDER BY cohort_week;
Golden Reference Ground Truth SQL (Runtime on 1M Rows: 148.60 ms)
-- Generate dynamic cohort retention matrix

SELECT date_trunc('week', u.signup_date) AS cohort_week,
       count(DISTINCT u.user_id) AS cohort_size,
       count(DISTINCT CASE
                          WHEN a.activity_date = u.signup_date THEN u.user_id
                      END) AS day_0_active,
       count(DISTINCT CASE
                          WHEN a.activity_date >= u.signup_date + 7
                               AND a.activity_date < u.signup_date + 14 THEN u.user_id
                      END) AS day_7_active,
       count(DISTINCT CASE
                          WHEN a.activity_date >= u.signup_date + 14
                               AND a.activity_date < u.signup_date + 21 THEN u.user_id
                      END) AS day_14_active
FROM app_users u
LEFT JOIN activity_log a ON u.user_id = a.user_id
GROUP BY cohort_week
ORDER BY cohort_week;
DuckDB Fixture Setup DDL & Data Population
CREATE TABLE app_users (user_id INT, signup_date DATE);


CREATE TABLE activity_log (user_id INT, activity_date DATE);


INSERT INTO app_users
VALUES (1, '2026-02-01'),
       (2, '2026-02-01'),
       (3, '2026-02-08');


INSERT INTO activity_log
VALUES (1, '2026-02-01'),
       (1, '2026-02-08'),
       (1, '2026-02-15'),
       (2, '2026-02-01'),
       (2, '2026-02-08'),
       (3, '2026-02-08');

5. Query Results & Expected Ground Truth

Tabular Comparison
Actual Candidate Output (14 rows × 5 cols)
# cohort_week cohort_size day_0_active day_7_active day_14_active
1 2025-12-29 00:00:00 44,448 44,448 0 0
2 2026-01-05 00:00:00 77,780 77,777 0 0
3 2026-01-12 00:00:00 77,777 77,777 0 0
4 2026-01-19 00:00:00 77,777 77,777 0 0
5 2026-01-26 00:00:00 77,779 77,779 2 1
6 2026-02-02 00:00:00 77,778 77,778 0 0
7 2026-02-09 00:00:00 77,777 77,777 0 0
8 2026-02-16 00:00:00 77,777 77,777 0 0
9 2026-02-23 00:00:00 77,777 77,777 0 0
10 2026-03-02 00:00:00 77,777 77,777 0 0
11 2026-03-09 00:00:00 77,777 77,777 0 0
12 2026-03-16 00:00:00 77,777 77,777 0 0
13 2026-03-23 00:00:00 77,777 77,777 0 0
14 2026-03-30 00:00:00 22,222 22,222 0 0
Expected Golden Output (14 rows × 5 cols)
# cohort_week cohort_size day_0_active day_7_active day_14_active
1 2025-12-29 00:00:00 44,448 44,448 0 0
2 2026-01-05 00:00:00 77,780 77,777 0 0
3 2026-01-12 00:00:00 77,777 77,777 0 0
4 2026-01-19 00:00:00 77,777 77,777 0 0
5 2026-01-26 00:00:00 77,779 77,779 2 1
6 2026-02-02 00:00:00 77,778 77,778 0 0
7 2026-02-09 00:00:00 77,777 77,777 0 0
8 2026-02-16 00:00:00 77,777 77,777 0 0
9 2026-02-23 00:00:00 77,777 77,777 0 0
10 2026-03-02 00:00:00 77,777 77,777 0 0
11 2026-03-09 00:00:00 77,777 77,777 0 0
12 2026-03-16 00:00:00 77,777 77,777 0 0
13 2026-03-23 00:00:00 77,777 77,777 0 0
14 2026-03-30 00:00:00 22,222 22,222 0 0
← Prev: Chronological Multi-Stage... ⊞ Back to Benchmark Dashboard Next: Exponential Time-Decayed ... →