E-Commerce Financial Aggregation without Join Fan-Out
1. Analytical Significance & What It Means
Scenario & Frontier TrapDemonstrates the Chasm Trap where joining multiple 1-to-many child tables (discounts and shipments) to an orders table causes row multiplication before aggregation, inflating financial sums.
Joining multiple 1:N child tables (order_discounts has 2 rows, order_shipments has 2 rows) directly to orders before GROUP BY creates an intermediate Cartesian fan-out of 2 * 2 = 4 rows per order. Aggregating over these inflated rows multiplies base_amount by 4x ($100.00 -> $400.00), total discounts by 2x ($15.00 -> $30.00), and shipping by 2x ($10.00 -> $20.00). Proper solution requires pre-aggregating child tables to 1 row per order in CTEs or correlated subqueries before joining.
Calculate total gross sales (sum of base_amount), total discount amount, and total shipping fees for each merchant. Ensure multiple discount line items and multiple shipment records for the same order do NOT multiply or artificially inflate the totals. Required output columns: - merchant_id (INT) - merchant_name (VARCHAR) - total_gross_sales (DECIMAL) - total_discount_amount (DECIMAL) - total_shipping_fees (DECIMAL) Row order does not matter.
2. Underlying Tables, Row Counts & Data Volume
4 Tables (1,003,000 rows • 23.0 MiB)Production-scale fixtures loaded into the isolated DuckDB sandbox with at least 1,000,000 records per table:
merchant_id (INTEGER) merchant_name (VARCHAR)| # | merchant_id | merchant_name |
|---|---|---|
| 1 | 1 | Acme Electronics |
| 2 | 1 | Acme Electronics |
| 3 | 2 | Acme Electronics |
| 4 | 3 | Acme Electronics |
| 5 | 4 | Acme Electronics |
| 6 | 5 | Acme Electronics |
| 7 | 6 | Acme Electronics |
| 8 | 7 | Acme Electronics |
| 9 | 8 | Acme Electronics |
| 10 | 9 | Acme Electronics |
| 11 | 10 | Acme Electronics |
| 12 | 11 | Acme Electronics |
| 13 | 12 | Acme Electronics |
| 14 | 13 | Acme Electronics |
| 15 | 14 | Acme Electronics |
discount_id (INTEGER) order_id (INTEGER) amount (DECIMAL(10,2))| # | discount_id | order_id | amount |
|---|---|---|---|
| 1 | 1 | 101 | 10 |
| 2 | 2 | 101 | 5 |
| 3 | 2 | 101 | 10.38 |
| 4 | 3 | 102 | 8.25 |
| 5 | 4 | 103 | 11.93 |
| 6 | 5 | 104 | 6.49 |
| 7 | 6 | 105 | 8.83 |
| 8 | 7 | 106 | 15.72 |
| 9 | 8 | 107 | 19.38 |
| 10 | 9 | 108 | 6.95 |
| 11 | 10 | 109 | 5.08 |
| 12 | 11 | 110 | 6.79 |
| 13 | 12 | 111 | 5.23 |
| 14 | 13 | 112 | 9.65 |
| 15 | 14 | 113 | 7 |
shipment_id (INTEGER) order_id (INTEGER) shipping_fee (DECIMAL(10,2))| # | shipment_id | order_id | shipping_fee |
|---|---|---|---|
| 1 | 1 | 101 | 4 |
| 2 | 2 | 101 | 6 |
| 3 | 2 | 101 | 5.78 |
| 4 | 3 | 102 | 4.7 |
| 5 | 4 | 103 | 5.82 |
| 6 | 5 | 104 | 5.72 |
| 7 | 6 | 105 | 5.41 |
| 8 | 7 | 106 | 3.97 |
| 9 | 8 | 107 | 5.28 |
| 10 | 9 | 108 | 6.59 |
| 11 | 10 | 109 | 4.01 |
| 12 | 11 | 110 | 6.39 |
| 13 | 12 | 111 | 6.13 |
| 14 | 13 | 112 | 3.91 |
| 15 | 14 | 113 | 5.28 |
order_id (INTEGER) merchant_id (INTEGER) base_amount (DECIMAL(10,2))| # | order_id | merchant_id | base_amount |
|---|---|---|---|
| 1 | 101 | 1 | 100 |
| 2 | 101 | 1 | 90.94 |
| 3 | 102 | 2 | 91.54 |
| 4 | 103 | 3 | 115.57 |
| 5 | 104 | 4 | 119.81 |
| 6 | 105 | 5 | 90.5 |
| 7 | 106 | 6 | 102.73 |
| 8 | 107 | 7 | 109.63 |
| 9 | 108 | 8 | 102.33 |
| 10 | 109 | 9 | 97.54 |
| 11 | 110 | 10 | 110.28 |
| 12 | 111 | 11 | 105.29 |
| 13 | 112 | 12 | 90.88 |
| 14 | 113 | 13 | 97.27 |
| 15 | 114 | 14 | 108.95 |
3. Graphical Query Execution Plan & Cost Bottlenecks
Physical Operator Tree (DuckDB 2.0 Profiler)Visual flowchart of the physical execution pipeline. Operators with high relative costs (≥ 20% of query runtime or heavy latency) are visually highlighted:
| Operator | Cost % | Latency | Processed Rows | Data Volume | Details / Table | Status |
|---|---|---|---|---|---|---|
| HASH_GROUP_BY | 86.0% | 309.58 ms | 999,999 | 49.84 MB | - | 🔥 BOTTLENECK |
| HASH_JOIN | 4.0% | 14.43 ms | 1,000,000 | 30.64 MB | - | Normal |
| HASH_JOIN | 3.1% | 11.25 ms | 1,000,000 | 42.20 MB | - | Normal |
| PROJECTION | 3.0% | 10.91 ms | 999,999 | 49.59 MB | Projections: ['merchant_id', 'total_gross_sales' | Normal |
| HASH_JOIN | 2.4% | 8.47 ms | 1,000 | 0.06 MB | - | Normal |
| TABLE_SCAN | 1.3% | 4.76 ms | 1,000,000 | 15.26 MB | memory.main.orders | Normal |
| PROJECTION | 0.1% | 0.19 ms | 1,000,000 | 42.20 MB | Projections: ['merchant_id', 'base_amount', 'dis | Normal |
| PROJECTION | 0.0% | 0.10 ms | 1,000 | 0.06 MB | Projections: ['merchant_id', 'merchant_name', 't | Normal |
| TABLE_SCAN | 0.0% | 0.02 ms | 1,000 | 0.02 MB | memory.main.merchants | Normal |
| PERFECT_HASH_GROUP_BY | 0.0% | 0.02 ms | 998 | 0.02 MB | - | Normal |
| PERFECT_HASH_GROUP_BY | 0.0% | 0.02 ms | 998 | 0.02 MB | - | Normal |
| TABLE_SCAN | 0.0% | 0.02 ms | 1,000 | 0.01 MB | memory.main.order_discounts | Normal |
| TABLE_SCAN | 0.0% | 0.01 ms | 1,000 | 0.01 MB | memory.main.order_shipments | Normal |
| PROJECTION | 0.0% | 0.01 ms | 1,000 | 0.01 MB | Projections: ['CAST(#0 AS SMALLINT)', '#1'] | Normal |
| PROJECTION | 0.0% | 0.00 ms | 998 | 0.02 MB | Projections: ['CAST(#0 AS INTEGER)', '#1'] | Normal |
| PROJECTION | 0.0% | 0.00 ms | 998 | 0.02 MB | Projections: ['CAST(#0 AS INTEGER)', '#1'] | Normal |
| PROJECTION | 0.0% | 0.00 ms | 1,000 | 0.01 MB | Projections: ['CAST(#0 AS SMALLINT)', '#1'] | Normal |
| PROJECTION | 0.0% | 0.00 ms | 1,000 | 0.01 MB | Projections: ['order_id', 'amount'] | Normal |
| PROJECTION | 0.0% | 0.00 ms | 1,000 | 0.01 MB | Projections: ['order_id', 'shipping_fee'] | Normal |
4. Queries Executed (Candidate vs Golden Ground Truth)
Beautified & Syntax HighlightedWITH order_level_discounts AS
(SELECT order_id,
sum(amount) AS discount_amount
FROM order_discounts
GROUP BY order_id),
order_level_shipments AS
(SELECT order_id,
sum(shipping_fee) AS shipping_fee
FROM order_shipments
GROUP BY order_id),
merchant_order_metrics AS
(SELECT o.merchant_id,
sum(o.base_amount) AS total_gross_sales,
coalesce(sum(d.discount_amount), 0.00) AS total_discount_amount,
coalesce(sum(s.shipping_fee), 0.00) AS total_shipping_fees
FROM orders o
LEFT JOIN order_level_discounts d ON o.order_id = d.order_id
LEFT JOIN order_level_shipments s ON o.order_id = s.order_id
GROUP BY o.merchant_id)
SELECT m.merchant_id,
m.merchant_name,
coalesce(mom.total_gross_sales, 0.00) AS total_gross_sales,
coalesce(mom.total_discount_amount, 0.00) AS total_discount_amount,
coalesce(mom.total_shipping_fees, 0.00) AS total_shipping_fees
FROM merchants m
LEFT JOIN merchant_order_metrics mom ON m.merchant_id = mom.merchant_id;
WITH agg_discounts AS
(SELECT order_id,
sum(amount) AS total_discount
FROM order_discounts
GROUP BY order_id),
agg_shipments AS
(SELECT order_id,
sum(shipping_fee) AS total_shipping
FROM order_shipments
GROUP BY order_id)
SELECT m.merchant_id,
m.merchant_name,
sum(o.base_amount) AS total_gross_sales,
coalesce(sum(d.total_discount), 0.00) AS total_discount_amount,
coalesce(sum(s.total_shipping), 0.00) AS total_shipping_fees
FROM merchants m
JOIN orders o ON m.merchant_id = o.merchant_id
LEFT JOIN agg_discounts d ON o.order_id = d.order_id
LEFT JOIN agg_shipments s ON o.order_id = s.order_id
GROUP BY m.merchant_id,
m.merchant_name;
CREATE TABLE merchants (merchant_id INT, merchant_name VARCHAR);
CREATE TABLE orders (order_id INT, merchant_id INT, base_amount decimal(10, 2));
CREATE TABLE order_discounts (discount_id INT, order_id INT, amount decimal(10, 2));
CREATE TABLE order_shipments (shipment_id INT, order_id INT, shipping_fee decimal(10, 2));
INSERT INTO merchants
VALUES (1, 'Acme Electronics');
INSERT INTO orders
VALUES (101, 1, 100.00);
-- 2 discounts applied to Order 101 ($10 and $5 = $15 total)
INSERT INTO order_discounts
VALUES (1, 101, 10.00),
(2, 101, 5.00);
-- 2 split shipments for Order 101 ($4 and $6 = $10 total)
INSERT INTO order_shipments
VALUES (1, 101, 4.00),
(2, 101, 6.00);
5. Query Results & Expected Ground Truth
Tabular Comparison| # | merchant_id | merchant_name | total_gross_sales | total_discount_amount | total_shipping_fees |
|---|---|---|---|---|---|
| 1 | 31 | Acme Electronics | 92.69 | 4.83 | 3.71 |
| 2 | 32 | Acme Electronics | 119.22 | 9.37 | 6.3 |
| 3 | 97 | Acme Electronics | 115.51 | 10.39 | 2.13 |
| 4 | 163 | Acme Electronics | 98.63 | 9.34 | 4.57 |
| 5 | 179 | Acme Electronics | 113.99 | 7.01 | 5.52 |
| 6 | 204 | Acme Electronics | 93.09 | 10.3 | 6.45 |
| 7 | 338 | Acme Electronics | 111.76 | 5 | 2.15 |
| 8 | 351 | Acme Electronics | 121.5 | 11.84 | 4.45 |
| 9 | 355 | Acme Electronics | 94.53 | 6.81 | 4.81 |
| 10 | 364 | Acme Electronics | 139.5 | 3.62 | 6.48 |
| 11 | 403 | Acme Electronics | 118.61 | 9.09 | 5.16 |
| 12 | 440 | Acme Electronics | 95.5 | 8.88 | 3.25 |
| 13 | 479 | Acme Electronics | 109.52 | 8.13 | 3.33 |
| 14 | 480 | Acme Electronics | 83.99 | 10.21 | 3.89 |
| 15 | 522 | Acme Electronics | 110.12 | 7.71 | 2.53 |
| 16 | 617 | Acme Electronics | 111.08 | 12.58 | 1 |
| 17 | 620 | Acme Electronics | 95.24 | 6.17 | 4.46 |
| 18 | 636 | Acme Electronics | 111.45 | 3.75 | 4.42 |
| 19 | 681 | Acme Electronics | 69.56 | 9.35 | 4.03 |
| 20 | 733 | Acme Electronics | 99.75 | 10.81 | 7.01 |
| 21 | 738 | Acme Electronics | 117.04 | 3.89 | 2.91 |
| 22 | 789 | Acme Electronics | 110.6 | 7.56 | 3.66 |
| 23 | 918 | Acme Electronics | 99.24 | 6.97 | 1.67 |
| 24 | 933 | Acme Electronics | 116.51 | 9.18 | 5.77 |
| 25 | 939 | Acme Electronics | 104.92 | 5.71 | 4.61 |
| # | merchant_id | merchant_name | total_gross_sales | total_discount_amount | total_shipping_fees |
|---|---|---|---|---|---|
| 1 | 2 | Acme Electronics | 91.54 | 8.25 | 4.7 |
| 2 | 5 | Acme Electronics | 90.5 | 8.83 | 5.41 |
| 3 | 9 | Acme Electronics | 97.54 | 5.08 | 4.01 |
| 4 | 10 | Acme Electronics | 110.28 | 6.79 | 6.39 |
| 5 | 19 | Acme Electronics | 90.41 | 8.24 | 4.4 |
| 6 | 20 | Acme Electronics | 111.25 | 9.58 | 3.42 |
| 7 | 22 | Acme Electronics | 87.27 | 6 | 4.06 |
| 8 | 24 | Acme Electronics | 107.25 | 14.07 | 6.7 |
| 9 | 27 | Acme Electronics | 96.76 | 14.68 | 4 |
| 10 | 42 | Acme Electronics | 117.51 | 7.85 | 1.4 |
| 11 | 44 | Acme Electronics | 97.93 | 8.3 | 5.22 |
| 12 | 49 | Acme Electronics | 88.7 | 7.37 | 5.28 |
| 13 | 53 | Acme Electronics | 87.56 | 4.77 | 4.09 |
| 14 | 79 | Acme Electronics | 110.47 | 7.95 | 2.86 |
| 15 | 84 | Acme Electronics | 84.88 | 8.09 | 4.64 |
| 16 | 88 | Acme Electronics | 83.44 | 10.31 | 7.32 |
| 17 | 89 | Acme Electronics | 110.14 | 8.73 | 6.36 |
| 18 | 98 | Acme Electronics | 122.46 | 7.25 | 4.66 |
| 19 | 106 | Acme Electronics | 91.53 | 2.66 | 4.61 |
| 20 | 117 | Acme Electronics | 104.09 | 9.72 | 5.57 |
| 21 | 125 | Acme Electronics | 104.44 | 7.99 | 3.13 |
| 22 | 137 | Acme Electronics | 74.95 | 5.93 | 6.2 |
| 23 | 140 | Acme Electronics | 109.34 | 2.21 | 6.6 |
| 24 | 141 | Acme Electronics | 100.44 | 12.53 | 4.92 |
| 25 | 145 | Acme Electronics | 117.68 | 9.2 | 3.45 |