Does a reviewer make a finance agent more reliable?

I had already turned a public financial report into an interactive explorer. I could compare periods and follow a number back to its page. The next question was whether an agent could choose the right evidence for a financial question, and whether adding a reviewer would improve the answer.

In this test, the single analyst did better. The separate reviewer added work and reduced the number of answers passing the checks. The failed outputs show where the added review changed a supported answer.

That sounds straightforward until you look at what can go wrong. Quarterly operating income increased by $86 million, while operating cash flow fell by $1,929 million. Both statements are correct. Explaining the cash movement as if it were the income movement would still produce a convincing paragraph with real numbers in it.

I used Codex and specialist agents to build this experiment. It uses public Novartis material and is independent demonstration work. It is not an internal company system or work I performed for Novartis.

Open the evidence desk. The calculated examples run online. The model comparison replays recorded calls so every visitor can inspect the same evidence.

Give the agent enough freedom to make a meaningful choice

I kept the existing dataset: 868 financial cells, including 115 explicitly unavailable values. I put it in a read-only SQLite database. Each fact retains its period, unit, accounting basis and source page.

The agent gets a catalog of available metrics, without their values. It can request figures, search eight reviewed report summaries using SQLite full-text search, or ask Python to calculate a comparison. It chooses which tools to use and which evidence to include in the answer. It cannot write arbitrary SQL, change the data or look elsewhere for a convenient explanation.

The search is deliberately small and lexical. It retrieves reviewed summaries, not every paragraph of the 54-page report. A failed search therefore tells us something about this small corpus, not whether the original report contains an answer somewhere else.

Keep the arithmetic out of the prose generator

The model selects references for numeric results. Python supplies the values from the observed facts or calculations. A missing prior-period cell stays missing. The comparison tool rejects a full-year amount paired with a quarterly amount. The model’s prose still needs its own review.

One useful example is free cash flow. Novartis defines it using operating cash flow less purchases of property, plant and equipment. Deducting intangible purchases as well would change that definition. The tool checks the arithmetic; the answer still has to explain which definition it used.

Another detail is the cash-flow table's combined lines. A movement labelled working capital and other operating cash-flow items should not be shortened into a claim about working capital alone. The number would survive that edit, but its meaning would change.

These checks do not make the prose automatically true. A valid citation can sit beside an unsupported cause. I kept exact value and reference checks separate from reviewing what the sentences actually say.

Give both review approaches the same starting point

I saved one initial analyst answer for each question. Two branches then received that exact draft and the evidence it had gathered. One branch had a separate reviewer role. The other asked the original analyst to take a second pass.

All three approaches use the same fixed model. Both review branches are fresh requests supplied with the saved draft and evidence. What changes is the role instruction, not the model's knowledge or access to another dataset.

Both branches received the same tools and additional budget: at most two model requests and six tool calls. The initial analyst had at most three requests. That extra control matters. If a reviewer improves an answer, I want to know whether the improvement comes from the different role or simply from another chance to inspect it.

If the first agent fails to produce a valid draft, both review branches are skipped and counted as unsuccessful answer opportunities. This version tests reviewing accepted drafts. It does not test whether another agent can rescue a malformed response.

The questions and expected results were frozen before the measured runs. They cover amounts, definitions, mixed arithmetic and explanation, unavailable figures, incompatible periods and an untrusted instruction embedded in a question. Gold numeric values and case-specific citation targets stay in the evaluator. The prompt does supply accounting guardrails, including period compatibility and the company’s free-cash-flow definition. This tests how the agent uses tools and evidence under those rules; it does not test whether it can discover the accounting rules itself.

This is a paired experiment on one report. The branches share a draft, so they are correlated observations. They are not three independent sets of financial work.

What the recorded runs showed

I ran twelve questions twice using nvidia/nemotron-3-super-120b-a12b:free, pinned to the Nvidia provider through OpenRouter. That produced 24 planned answers for each approach. The single analyst passed every exact reference, quantity and status check on 16 of 24. The analyst/reviewer pair passed 14 of 24. The analyst taking a second pass passed 12 of 24.

The initial analyst made 65 model requests. The separate reviewer added 38; self-review added 37. Because both branches reuse the same draft, the experiment made 140 physical model requests, not the sum of three independent workflows. Every measured request returned a cost of $0.

The median observed duration was about 34 seconds for the first answer, 69 seconds including the reviewer and 71 seconds including self-review. These durations include local quota pacing and provider time. They are not an isolated inference-speed benchmark.

An independent coding agent also checked every headline, claim, result and limitation against the sources using the frozen rubric. That model-assisted source audit passed 12 of 24 single-analyst answers, 9 of 24 reviewer answers and 9 of 24 self-review answers. It is a separate review with inspectable notes, not human financial assurance or a universal accuracy score.

Before the measured run, four practice requests to a free Gemma endpoint were rate-limited. I changed to Nvidia before starting the fixed comparison and retained the practice records. Those four rejections returned no usage amount, so the total including development is recorded as unknown rather than inventing a zero. Every request was restricted to zero-priced routing, with no paid fallback.

What changed in two answers

In the second run of the SG&A question, Python calculated a signed change of +61 USD million from −3,440 − (−3,501). The initial headline said the expense "changed by +61 USD million". The reviewer changed that to "increased by +61 USD million". The published expense amounts became less negative, which means the expense was lower. The number and its references still passed the exact checks, but the new headline failed the separate source review.

In another second run, the analyst correctly compared full-year IFRS net income of 13,967 with 11,939 USD million, an increase of 2,028 USD million, using the report's income statement. The reviewer received those same current, prior and change calculations. It then returned "Unable to verify change calculation", with the limitation "Cannot produce change result without confirmed calculation tool output." The output it said it needed was already in its inherited evidence. The branch returned an insufficient-evidence answer in place of a supported comparison.

The next version would keep a validated parent answer available when a review branch fails. I would improve recovery from invalid outputs and verify that the final sentences preserve the chosen values, signs and source scope before adding more roles. This bounded setup used one model and short request budgets. It does not establish that review generally harms answers.

Follow a question through the implementation

The interface exposes the actual SQL, bound parameters, returned rows and calculated results. Evidence-backed claims link to source figures, summaries or calculations. Statements about unavailable evidence can instead explain the boundary without pretending to have a supporting figure. A reviewer can follow the working without trusting the final paragraph first.

You can run the same comparison tool directly. From the Python project folder, after installing its requirements:

from backend.finance_tools import FinanceTools
 
with FinanceTools() as tools:
    result = tools.execute("calculate_comparison", {
        "kind": "metric_change",
        "section": "income",
        "metric_id": "operating_income",
        "current_period": "2025-Q4",
        "prior_period": "2024-Q4",
    })

This reads the two source amounts and returns the $86 million change with its input references. No model or key is involved. The agent's job is to choose this operation when it fits the question, then explain the result without changing its meaning.

The online service is a real Python/FastAPI function behind this Next.js website. The SQL and calculated examples run there. The recorded agent experiment runs locally against a fixed free model through OpenRouter. The key stays outside the repository and is never deployed. Public model calls are disabled in code.

That separation keeps the page useful when a free provider is busy. A visitor can still inspect the report and replay the recorded experiment, including unsuccessful attempts.