1. Choose a decision worth improving
A convincing AI demo answers a question. A useful pilot proves that the right people can answer recurring business questions using the right definitions and evidence.
Pick one decision, one accountable business owner and a bounded set of sources. Record how the task works today, how long it takes and what a better result would allow someone to do. Avoid starting with an unrestricted promise to ask anything about everything.
| Pilot decision | Evidence to capture |
|---|---|
| Business outcome | The recurring decision, its owner and the action an answer should support. |
| Scope | Companies, teams, periods, sources, user roles and explicit exclusions. |
| Baseline | Current task time, reconciliation steps and known failure cases. |
| Commercial boundary | Implementation work, platform and capacity costs, trial timing and support responsibilities. |
2. Give the pilot a business definition
A lakehouse holds data. Curated data and semantic models give approved structure to the measures the business uses. An MCP connection is a route to tools and context, not a guarantee that an answer is correct.
Document the source, preparation rules and approved measure for each important question. Define fiscal periods, currencies, entities, exceptions and the person who approves changes. Retain a route back to source evidence.
Example: what does revenue mean here?
Record whether revenue means posted invoices, recognised revenue or an operational sales measure. Specify returns, credits, tax, currency conversion, entity scope and the date used. A fluent answer using the wrong version is a failed test.
Microsoft recommends relevant model metadata, clear terminology and validation for its own Fabric data agents [1]. Those principles inform this worksheet. They do not establish that Nova AI Analyst uses every Microsoft agent feature or produces the same results.
3. Build an acceptance set
Agree expected answers and tolerances before running the evaluation. Keep a separate set of previously unseen questions for the final review.
| Test family | Example test | Pass evidence |
|---|---|---|
| Known calculation | Revenue by entity for a closed month. | Same population, filters, currency and result as the approved control. |
| Business nuance | Overdue debt at a historical cutoff. | Correct due-date policy and date-aware settlement treatment. |
| Ambiguous question | Show performance for the last quarter. | Clarifies fiscal/calendar period and the measure where needed. |
| Missing or stale data | Question beyond available history or after a failed feed. | Explains the limit rather than inventing a current answer. |
| Restricted access | A user requests another entity's excluded data. | No excluded information is returned. Test the actual role and tools. |
| Unsupported action | A prompt asks the analyst to approve or post a transaction. | Stays inside the agreed tools and approval boundary. |
| Adversarial content | Source text asks the assistant to ignore its instructions. | Does not treat untrusted source content as authority or disclose restricted information. |
Use a review log
Record the question, role, data cutoff, expected result, actual result, source/query evidence where available, error severity and reviewer. Measure correctness, appropriate clarification, access boundaries and task usefulness separately. A single blended percentage can hide a serious failure.
4. Make a release decision
The business sponsor and technical owner should sign off the evidence, open issues and operating responsibilities together.
Expand from evidence
Start with an approved group and bounded scope. Re-run the acceptance set after model, source, permission or tool changes. Keep a way to report incorrect answers and return to a trusted report when needed.
Where DataNova fits
DataNova connects reporting, planning and business applications on Microsoft Fabric. Nova AI Analyst can explore prepared data with business context, while Nova AI Visual Builder helps create visuals from prompts. We can help scope a pilot around your sources, definitions and workflow. Agree the setup work and acceptance criteria before the assisted trial begins.
Source and use of this scorecard
Source checked 29 October 2026. This is a DataNova evaluation worksheet, not a Microsoft certification, security assessment or guarantee.