Find the budget, visit and invoice problems while there's still time to fix them.

1. The problem: found too late

Under Support at Home, each participant has a quarterly budget. Unspent money above the carry-over limit is lost to the participant, and overspend has to be absorbed or clawed back. Both are usually discovered after the quarter closes, when nothing can be done. The same data also hides missed and late visits and errors in third-party provider invoices.

Approach. Join the three systems that each hold part of the picture (rostering, the worker app's check-ins and provider invoices) into one lakehouse, and answer four questions every morning. The budget forecast isn't a straight line: it's spend to date + in-house visits already booked for the rest of the quarter + provider invoices at their current run rate.

For scale: in the quarter that just closed, the same rule estimates $366k was lost to participants. The difference this time is that the $311k is flagged in October, with ten weeks left to act.

2. An open lakehouse: ADLS stores, Fabric computes

Approach. Azure Data Lake Storage Gen2 is the system of record. Every layer (raw files, then bronze, silver and gold Delta tables) lives in your own storage account. Fabric reaches it through OneLake shortcuts and does the compute. Pause or remove Fabric and nothing is lost, and Databricks, Synapse, DuckDB or Polars can read the same tables.

What's in it

• ADLS Gen2 (hierarchical namespace, TLS 1.2, no public blob access), Australia East, with two containers: raw (landing zone, files as received) and lake (Delta tables plus the pipeline SQL).
• Fabric F2 capacity, created paused and resumed only for runs. Workspace, lakehouse lh_care, notebook nb_care_medallion, daily 06:00 AEST schedule.
• OneLake shortcuts: Files/raw and Files/lake point at ADLS. Tables/gold_* shortcuts expose gold to the Fabric SQL endpoint without copying data.
• No Power BI. The dashboard is a Streamlit app querying gold Delta directly with DuckDB, signed in with Entra ID.
• Data stays onshore: storage and capacity both in Australia East.

Demonstrable artifacts

• infra/main.bicep: storage account, containers, RBAC and the (optional) Fabric capacity
• infra/deploy.ps1: deploys the Bicep and pauses the capacity immediately
• infra/upload-data.ps1: lands raw files and pipeline SQL with an Entra login, with no keys
• fabric/deploy-fabric.ps1: Fabric REST API for the workspace, lakehouse, connections, shortcuts, notebook, run and schedule

3. Medallion pipeline, checked against a known answer

Approach. The transform logic is plain Spark SQL files in transform/sql, run in order. Bronze keeps raw data as Delta. Silver cleans it: AU dd/MM/yyyy dates, de-duplicated app events, check-ins matched to bookings. Gold answers the four questions. The synthetic generator plants specific problems, and the gold tables are checked against them.

1. Raw: 124 CSV / JSONL files
2. Bronze: 7 tables, as received
3. Silver: 9 cleaned, conformed tables
4. Gold: 4 tables, one per question
5. Answers: budgets, visits, workers, invoices

Planted problem → did the pipeline find it?

The forecast, in the SQL itself

-- transform/sql/gold/01_budget_utilisation.sql (excerpt)
visit_spend_aud + invoice_spend_aud + booked_aud
  + invoice_spend_aud / days_elapsed * (days_in_quarter - days_elapsed)  AS forecast_spend_aud
...
CASE
  WHEN forecast_spend_aud > budget_aud       THEN 'Over budget'
  WHEN forecast_spend_aud < 0.7 * budget_aud THEN 'Under-used'
  ELSE 'On track'
END AS budget_status,
-- illustrative carry-over: unspent funds up to the greater of $1,000 or 10% of budget
GREATEST(budget_aud - forecast_spend_aud - GREATEST(1000.0, 0.1 * budget_aud), 0.0) AS est_funds_lost_aud

Demonstrable artifacts

• datagen/generate.py: synthetic source files with the planted problems (seeded, standard library only)
• transform/bronze_sources.json: which raw paths load into which bronze table
• transform/sql/silver: 9 cleaning and conforming steps
• transform/sql/gold: the 4 gold tables, one per question
• fabric/nb_care_medallion.py: the Fabric notebook (Git source format) that runs the SQL on Spark

4. Interactive demo: the gold tables

Approach. These charts are drawn in your browser from the pipeline's actual gold output (exported to a small JSON file, as of 18 Oct 2026). Pick a region; hover any mark for its numbers. The production dashboard is the Streamlit app further down, which queries the same tables live with DuckDB.

The production dashboard: Streamlit + DuckDB

A code-first app (app/dashboard.py) with KPI tiles, the budget distribution, funds at risk by region and a ranked action list, plus Visits, Workers and Invoices tabs. With CARE_SOURCE=adls it reads gold Delta straight from ADLS (delta_scan('abfss://lake@…/gold/…')) using the viewer's Entra login, with no gateway, extract or refresh schedule.

5. Same SQL, two engines: test on a laptop, run on Fabric

Approach. The Spark SQL files are the single source of transform logic. In Fabric, the notebook reads them from lake/_pipeline/sql and runs them on Spark. On a laptop, transform/run_local.py transpiles the same files to DuckDB with sqlglot and runs them in seconds, with no cloud cost, before anything ships.

1. transform/sql: 13 Spark SQL files
2. Laptop: sqlglot → DuckDB, all 13 steps in ~1 second
3. Local Parquet: for testing and review
4. Fabric Spark: notebook, scheduled daily
5. ADLS lake: Delta, the system of record

# transform/run_local.py (core loop)
for f in sorted((SQL / layer).glob("*.sql")):
    spark_sql = f.read_text().replace("{{AS_OF_DATE}}", a.as_of)
    duck_sql = sqlglot.transpile(spark_sql, read="spark", write="duckdb")[0]
    ...
    con.execute(f"COPY {table} TO '{out.as_posix()}' (FORMAT parquet)")

Two portability lessons are now in the SQL: use TIMESTAMPDIFF(SECOND, …) rather than UNIX_TIMESTAMP arithmetic, and wrap date-plus-interval in CAST(… AS DATE), because DuckDB returns a timestamp there.

Demonstrable artifacts

• transform/run_local.py: transpiles and runs every step on DuckDB, writing Parquet
• transform/sql: the single source of transform logic, shared by both engines
• fabric/nb_care_medallion.py: the Spark side, reading the same files from lake/_pipeline/sql

6. Keyless identity, and an honest status

Approach. Everything authenticates with Entra ID. People upload data and read the dashboard with their own login (az login, then DuckDB's credential_chain). Fabric reaches ADLS with its workspace identity, granted Storage Blob Data Contributor through RBAC. Account keys exist only as a fallback flag (-AuthMode Key).

1. You: Entra sign-in, no keys
2. Entra ID: issues tokens, RBAC on storage
3. ADLS: upload and read working
4. Dashboard: DuckDB → ADLS verified
5. Fabric: token refused pending

Where it's up to

The Fabric run is waiting on one tenant-level switch. Fabric depends on the Power BI Service enterprise app (even though this design uses no Power BI reports), and in the test tenant that app is disabled. An Entra administrator enables it, someone signs in once at app.fabric.microsoft.com, and fabric/deploy-fabric.ps1 -Schedule does the rest. Workspace creation can stay locked down to named people in Fabric's tenant settings.

Demonstrable artifacts

• infra/main.bicep: RBAC role assignment for the deploying user; storage with public blob access off
• fabric/deploy-fabric.ps1: provisions the workspace identity, grants it storage RBAC, creates the ADLS connections
• app/dashboard.py: DuckDB credential_chain with the Azure CLI login, no stored keys
• infra/capacity.ps1: status / resume / suspend the capacity

7. What it can be used for

The POC answers four questions for one program, but its parts are general: land files from systems you don't control, keep them in your own storage, clean them with SQL you can test on a laptop, and put a ranked action list in front of the people who can act on it.

From POC to production

• Swap synthetic files for real extracts on the same folder contract: Data Factory, Eventstream for the mobile app, or SFTP → raw/.
• Incremental loads with Delta MERGE on booking, event and invoice keys instead of full overwrite.
• Data-quality expectations as a gold table (row counts, rejected rows), shown in the dashboard.
• Purview scanning for lineage and classification once real personal information lands.
• Dashboard on Azure Container Apps behind Entra ID (Easy Auth).

Capability scorecard

Cost