For a finance agent, document extraction is not the product. It is the load-bearing part underneath the product. So the buy-vs-build question is not just whether a parser can read a page. It is whether the parser fits the workflow you plan to ship. We tested that two ways: first on invoice extraction, where the vendors are already very close, and then inside a six-turn contract-review agent, where the question changes, the evidence has to be defended, and cost has to be counted outside the controller's LLM bill.
Start with the shape of the work. Some work should be split by task and handed to a permanent specialist. It repeats. It gets measured the same way every day. The specialist gets better because the job stays still long enough for compounding to matter.
Other work has to be split across time. A finance agent does not know its next move until the last one lands. It reads, reasons, retrieves, checks a clause, follows a number into an exhibit, changes direction, and only then knows what evidence it needs. That layer should stay inside the agent. It is adaptive by design.
Document parsing is the first kind of work. It is task-sliced. Reading documents is one narrow job, done forever, across scans, tables, statements, order forms, invoices and contracts your team has not seen yet. A parser vendor has lived inside that job for years. Your finance agent is the second kind. It is time-sliced. It switches by the turn because the question changes as the conversation unfolds.
That is the build-vs-buy answer. Buy the parser. Build the agent. Most mistakes in this space come from applying the wrong slicing to the wrong layer.
The reason to buy is not that parsing is mystical. It is that this is a specialist part. Nobody buys a jacket because they understand the zipper mechanism. They care because a good jacket with a dead zipper is useless, and a trusted name on the pull means the zipper will not be the reason the jacket dies. Document extraction is that kind of part. It is not the product, but when it fails, the product fails with it.
Which leaves the real decision: which vendor. We put three through the same finance work, with the same agent and the same grader. We assumed a leaderboard would sort them. What we got instead was a more useful answer. It came from measuring them two different ways and looking at the distance between the results.
Here is the pipeline we built and threw away. A contract arrives. You call Gemini to draft a JSON schema for it. You validate that schema. You hand the schema to a parser and ask it to extract against it. You repair whatever comes back. It demos beautifully. It benchmarks respectably. And it is often the wrong default for an open-ended agent, for a reason no benchmark score will show you.
A schema pre-assigns the operations. That works when the job is task-sliced and known in advance. Pull invoice number, vendor, date, subtotal, total. Fine. Name the slots. Run the extractor. Check the values.
A contract agent is different. It does not know what it will ask next. A renewal question leads to a notice-window question, which leads to a termination-penalty question, which sends the agent into an exhibit nobody planned to read. Four turns in, the agent needs a fact that did not exist in the schema. Now the schema is not a convenience. It is a guess you have to maintain forever.
That is the trap. You may not have removed the engineering work you were trying to buy your way out of. You may have moved it, from parsing the document to maintaining a schema library, and a schema library rots in exactly the way document layouts rot. Every new contract shape can become a ticket. Every new question can become another slot.
The alternative is boring to describe and hard to buy. Parse the document once. Then ask it anything, mid-conversation, as the reasoning unfolds, without a pre-written schema and without a second model in the loop. One hop, not two. That is the time-sliced operating model, and it is the axis the usual leaderboard does not have a column for.
Because the easy answer and the production answer are not the same. Claude and Gemini can read PDFs directly. On clean pages, they can look excellent. On dense financial tables, scanned receipts, nested schedules and contract exhibits, the question is not whether they can sometimes see the value. They can. The question is whether they see it reliably enough to sit underneath a finance workflow.
We did run that baseline for the contract workflow, and the result deserves caution. After tightening only the turn intent, Reducto schemaless returned 428 of 435 engine-evidence terms. Gemini native PDF returned 426 of 435. Claude native PDF was not close in this setup, at 327 of 435. The takeaway is not that a native model owns the category. It is that native-PDF baselines are now strong enough that a parser benchmark has to control very carefully for context, schema shape, history and scoring.
The deeper answer is provenance and operating model. A finance answer has to be defensible. When an agent says the termination penalty is $40k, someone needs to ask where that came from and get a page, a clause, a cell, or a coordinate. In this setup, Reducto exposed document-engine controls: parse, split, extract and cite. Gemini exposed a native-PDF session: upload once, ask follow-up extraction questions, and let the model manage retrieval inside that session. Both are useful. They are not the same product boundary.
A parser can be graded on its own work, or on the work it enables. Those are different questions, they need different instruments, and almost every benchmark you will read picks one and calls it the answer. We ran both, deliberately, and that turned out to be the whole point.
Method 1 · the parser alone. No agent. We read each parser's raw output and check, value by value, whether the invoice's real numbers are present and correctly placed. Deterministic: a value is in the output or it is not. No LLM sits in the loop, so the score does not move between runs and it belongs to the parser and nothing else.
Applied to: 19 invoices and bills. Clean, scanned OCR-only, and dense multi-page.
Method 2 · the parser inside an agent. A fixed controller agent (Claude Sonnet 4.6, identical for every parser) works a six-turn contract review. Each turn asks the document engine for focused evidence, then a deterministic scorer checks both the engine evidence and the final answer against hand-built gold facts. Realistic, and noisier, because the agent is now part of what you are measuring.
Applied to: 10 commercial contracts. MSAs, order forms, sales orders, estimates.
All three parsers found essentially every value on essentially every invoice. Field accuracy across 19 documents: Reducto 0.9812, Datalab 0.9803, LlamaParse 0.9737. Three quarters of one point separates first from last. There is no story here, and that is the story.
Per-field pass rate across the invoice set. Number, vendor, date, currency, subtotal and total are captured by all three parsers on every single invoice. The one crack in the entire grid is LlamaParse dropping the single line item on one scanned receipt. That is the sum total of the field-level disagreement between three competing commercial products.
It holds under difficulty, too. On clean invoices all three score 1.00. On scanned, image-only PDFs that have to be OCR'd, Reducto and Datalab still score 1.00 and LlamaParse 0.97. Only the dense multi-page bills bite, and they bite everyone equally: 0.93, 0.93, 0.93. Not one of those is a vendor difference. They are all the same handful of buried needle values and one nested subtotal that nobody gets.
If you are choosing a parser to read invoices, stop reading benchmarks and choose on price. Field capture is a solved problem and everyone has solved it.
Capture is not placement. A total sitting in a wall of loose text is technically captured and practically useless, because the agent downstream has to guess which row it belongs to. Here the parsers finally separate, and by eleven points.
Structural fidelity. How often a value landed inside the correct table cell instead of floating in text. Reducto 0.89, LlamaParse 0.83, Datalab 0.78. Datalab more often drops totals into plain prose. This is a real, measured, eleven-point gap, and it is the only place invoices sort the vendors at all.
Hold on to that number. It is the strongest reason to believe Reducto should win the contract test, and it is about to fail to predict anything.
Parse cost against parse time, per document. Reducto is 6.6× faster than Datalab (5.0s vs 41.5s) and 5.5× more expensive ($0.165 vs $0.040). There is no free lunch and the lunch is priced exactly as you would expect. *Datalab's rate is extrapolated from its published schema-extract price and LlamaParse's is list price, because neither API returned billed spend. Only Reducto's figure is metered.
Five seconds versus forty-one is not a rounding error. If the parse sits in front of a user, or inside a turn the agent has to complete before it can ask the next question, that is the difference between a product and a batch job. Reducto is selling speed and structure, and the price tag is honest about it.
A controller does not ask for a PDF parse. A controller asks whether a contract is safe to renew, what cash is at risk, which clauses need renegotiation, and what has to be calendared before the notice window closes. So the benchmark follows the conversation: six turns, one fixed controller, one document engine call per turn, and a deterministic score against 435 gold facts.
Benchmark architecture. The custom finance agent is fixed across vendors. The evidence request adapter is the thin glue layer: it turns the current finance turn into the provider's natural call, whether that is schemaless extraction, a schema-based request, split/extract, or a native-PDF follow-up. It records cost and latency, then sends cited evidence back to the agent without moving finance reasoning out of the controller.
There are two honest ways to run that workflow. The first is the no-schema agentic path: schemaless, turn-by-turn extraction. The controller asks for the evidence it needs now. The document engine retrieves focused evidence. The controller answers. No schema planner. No hand-written field contract for every future question.
The second is useful, but different: schema-assisted extraction. A schema-planner agent chooses a JSON schema for the current turn, then the document engine extracts into that shape. That gives Datalab and LlamaExtract a fairer operating mode, and it gives Reducto a structured variant too. But it is not the same product surface. It adds a schema planning hop, a schema library, and a new failure mode.
| Primary no-schema/native setup | Schema? | Docs | Final accuracy | Final term recall | Engine evidence recall | Engine terms | Context management for an agent |
|---|---|---|---|---|---|---|---|
| Reducto schemaless | No | 10 | 95.41% | 97.70% | 98.39% | 428/435 | Parse once, route pages with Split, extract focused evidence each turn, cite pages, keep the controller out of full-PDF context. |
| Gemini native PDF | No | 10 | 92.21% | 94.25% | 97.93% | 426/435 | Upload once and ask follow-ups in-session. The model owns PDF reading, retrieval and context handling. |
| Claude native PDF | No | 10 | 79.96% | 84.37% | 75.17% | 327/435 | Same native-PDF session pattern, but lower extraction recall in this run. |
Main contract workflow, refreshed scorer. Ten contracts, six turns, 435 graded terms. Turn weights are 10%, 15%, 17%, 22%, 16%, and 20% for commercial status, renewal, cost exposure, SLA/liability/security, data obligations, and action summary. All providers were rerun for the clarified Turn 6. Gemini and Claude use native-PDF session runs where the PDF is uploaded once and each turn asks a follow-up extraction question inside the same document session.
The table is not a single-score ranking. Gemini is much cheaper and faster in this clean-contract run. Reducto has the highest deterministic final score and the highest engine-evidence recall. Claude trails both on this setup. The important point is the shape of the workflow, not a trophy line.
The main story is whether the engine can support changing contract questions without a schema. The spider chart keeps the tradeoffs visible: quality, speed, cost, token load and context control do not move together.
No-schema/native decision spider. Scale is 0 to 1, further out is better. Accuracy and evidence recall use the deterministic summary table in the appendix. Latency, total cost and token load are normalized as best measured value divided by this provider's measured value, so lower burden sits farther out. Context control is a disclosed workflow rubric: Reducto schemaless scores 1.00 for explicit parse/split/extract handles; Gemini and Claude score 0.62 because retrieval stays inside the native-PDF model session. Read this as decision shape, not a universal ranking.
| Turn | What the controller asked | Weight | Reducto evidence checks | Gemini evidence checks | Claude evidence checks |
|---|---|---|---|---|---|
| 1 | Basic commercial status | 10% | 200/200, 100.0% | 200/200, 100.0% | 200/200, 100.0% |
| 2 | Renewal and termination | 15% | 118/118, 100.0% | 112/118, 94.9% | 94/118, 79.7% |
| 3 | Financial exposure | 17% | 176/180, 97.8% | 180/180, 100.0% | 86/180, 47.8% |
| 4 | Service, liability and support risk | 22% | 112/112, 100.0% | 112/112, 100.0% | 82/112, 73.2% |
| 5 | Data, audit, confidentiality and law | 16% | 92/92, 100.0% | 92/92, 100.0% | 88/92, 95.7% |
| 6 | Controller action summary evidence | 20% | 160/168, 95.2% | 156/168, 92.9% | 118/168, 70.2% |
After the Turn 6 clarification across providers, Reducto leads the headline engine-evidence view: 428/435. Gemini is close at 426/435. That is why this is a workflow result, not a slogan. Reducto gives explicit document-engine controls. Gemini shows that a native-PDF session can be very competitive on clean contract packets.
The remaining gap is partly final synthesis. Reducto's engine returned 160/168 Turn 6 evidence checks, but the final controller answer preserved 146/168. Gemini showed an even larger Turn 6 engine-to-final gap: 156/168 evidence checks became 120/168 final checks. That matters for product design: the document engine can be strong and the controller can still drop facts while writing a summary. We score both surfaces so that failure is visible.
We also tested the schema path because many teams will try it. This is not the main story, but it is not a toy either. Datalab and LlamaExtract are strongest when you give them a schema, and Reducto can be run that way too. So we ran the same 10 contracts, the same six turns, the same 435-term gold, but added a schema-planner step before each extraction.
| Schema-assisted setup | Schema? | Docs | Final controller | Term recall | Terms | Median latency | Engine cost | Agent cost | Total |
|---|---|---|---|---|---|---|---|---|---|
| Reducto schema | Yes | 10 | 94.94% | 97.01% | 422/435 | 992s | $41.79 | $5.10 | $46.89 |
| LlamaExtract schema | Yes | 10 | 94.30% | 96.09% | 418/435 | 609s | $14.04* | $4.31 | $18.35* |
| Datalab segment schema | Yes | 10 | 89.66% | 91.03% | 396/435 | 593s | $18.72* | $4.17 | $22.89* |
Schema-assisted contract run. Same 10 current contracts and same 435-term scorer. The schema is chosen dynamically per turn by a schema-planner agent. Asterisks mark provider costs modeled from public credit or processor page rates rather than direct metered billing in the API response. Datalab balanced extraction may also carry a variable compute surcharge; no unambiguous surcharge was available in these logs.
Schema-assisted decision spider. Scale is 0 to 1, further out is better. Accuracy and evidence recall use the current 435-term run. Latency and total cost are normalized as best measured value divided by this provider's measured value. Schema burden is a workflow rubric: all three require the schema-planner hop and score 0.65. Context control is disclosed as Reducto 0.92, Datalab 0.72 and LlamaExtract 0.70, based on explicit document-engine handles and inspectable retrieval.
The schema-assisted result is useful, but it changes the question. Reducto schema has the highest deterministic final score and engine-evidence recall in this slice. Datalab has the lowest median latency. LlamaExtract has the lowest modeled total cost and sits close to Reducto on deterministic quality after the Turn 6 full-document route change. The product point is that all three are helped by a planning layer. That is fair for teams that already know their fields. It is less natural for a controller agent whose next question is discovered mid-conversation.
Once quality is close, the tiebreak is whatever else you have to live with. For this use case that means money, latency, token load, context control and schema work.
| Main no-schema/native setup | Document-engine units | Engine cost | Agent cost | Measured run total | Median latency | What you are buying |
|---|---|---|---|---|---|---|
| Reducto schemaless | 2,782 split/extract credits; parse cache hit | $41.73 | $5.41 | $47.14 | 908s | explicit parse, split and extract controls with highest evidence recall |
| Gemini native PDF | 506,559 input tokens + 87,580 output tokens | $0.37 | $3.63 | $4.01 | 323s | low measured cost and latency; strong native-PDF baseline on clean contracts |
| Claude native PDF | 2,497,932 input tokens + 109,199 output tokens | $9.13 | $3.75 | $12.88 | 373s | native-PDF session pattern, but weaker extraction recall in this run |
Reducto's cost needs a careful label. The fresh run reused persisted parse jobs, so the table shows the interactive Split + Extract cost across six turns. Reducto credits are priced at $0.015 per credit*. Any first-session parse cost should be added separately if the parse job is not already cached.
The cost of schemaless extraction is not just the API bill. It changes where complexity lives. The controller can ask a new question mid-conversation without first choosing, generating, validating or repairing a schema. That can be worth paying for. It can also be too expensive for a clean, low-risk workflow where a native-PDF session already performs well.
*Pricing note. We used public list prices so the comparison is reproducible. Real customer bills may differ with volume discounts, committed-use pricing, free credits, negotiated contracts, taxes, and provider-specific metering rules.
Buy the parser, but benchmark the workflow you actually plan to ship. A field-extraction leaderboard is not enough for an agentic finance product.
Invoice extraction is mature. In our runs, Reducto, Datalab and LlamaParse all found essentially the same fields. Contracts were different. The parser was no longer just reading a page. It was sitting inside a finance conversation, answering a different evidence request on every turn.
That is where the operating model mattered.
Schemas worked when the question was known in advance. Native-PDF sessions were surprisingly strong on clean contracts. Reducto's schemaless workflow gave us the most inspectable path: parse once, route pages, extract focused evidence, cite, repeat.
The next benchmark should be harder: amendments, scanned attachments, conflicting exhibits, and multi-page tables. But the lesson already holds. Do not buy a leaderboard. Buy the document boundary your agent can actually live with.
The findings are above. This appendix defines the terms, shows exactly how a score is built, and lists what to trust and what not to. If a chart confused you, this is where to come back to.
No assistant. We read each parser's own output and check, value by value, whether the invoice's real numbers and text are present and correctly placed. Nothing else touches it, so the score is the parser's alone, and it is deterministic: a value is in the output or it is not. It does not move run to run.
Field accuracy is the share of the invoice's real values found anywhere in the parser's output. Structural fidelity is the share that landed inside the correct table cell rather than floating in loose text. The second is the one that matters and the only one that separated anybody.
A fixed controller (the same Claude Sonnet 4.6 for every parser) writes the final answer, so any difference is the document engine and not the assistant. The current headline contract run is no-schema or native-PDF: for each turn, the controller asks for focused evidence, the document engine extracts that evidence, and the controller answers the user's finance question.
Reducto uses parse once, then Split + Extract on every turn. Gemini and Claude use native PDF sessions: the PDF is uploaded once, then each turn asks a fresh extraction question in the same document session. We keep document-engine spend separate from controller-agent spend, so parser cost does not get hidden inside the agent bill.
We also show a schema-assisted bake-off on the same 10 contracts and 435-term scorer. That test is useful, but different: the engine receives stronger structure through a schema-planner step. It belongs in the workflow comparison, not in the same leaderboard as the no-schema/native run.
Scoring is deterministic against hand-built gold facts. We score two surfaces where available: engine evidence, which checks what the document engine returned, and final controller, which checks what made it into the answer the user would see. Equivalent date, money and name formats are normalized before matching; a contradicted value gets no credit.
Every contract session is graded 0% to 100%. The score is weighted toward the turns a controller would care about most when money, service risk and next actions are on the line:
Every headline number is built in three stages: turn → session → cell.
The contract tables below are copied from the regenerated deterministic summary CSV. They keep the two operating models separate and keep document-engine/native-PDF cost separate from the controller LLM cost.
| No-schema/native setup | Docs | Final accuracy | Term recall | Terms | Engine evidence recall | Engine terms | Median latency | Engine cost | Agent cost | Total |
|---|---|---|---|---|---|---|---|---|---|---|
| Reducto schemaless | 10 | 95.41% | 97.70% | 425/435 | 98.39% | 428/435 | 908s | $41.73 | $5.41 | $47.14 |
| Gemini native PDF | 10 | 92.21% | 94.25% | 410/435 | 97.93% | 426/435 | 323s | $0.37 | $3.63 | $4.01 |
| Claude native PDF | 10 | 79.96% | 84.37% | 367/435 | 75.17% | 327/435 | 373s | $9.13 | $3.75 | $12.88 |
| Schema-assisted setup | Docs | Final accuracy | Term recall | Terms | Engine evidence recall | Engine terms | Median latency | Engine cost | Agent cost | Total |
|---|---|---|---|---|---|---|---|---|---|---|
| Reducto schema | 10 | 94.94% | 97.01% | 422/435 | 99.08% | 431/435 | 992s | $41.79 | $5.10 | $46.89 |
| LlamaExtract schema | 10 | 94.30% | 96.09% | 418/435 | 96.55% | 420/435 | 609s | $14.04* | $4.31 | $18.35* |
| Datalab segment schema | 10 | 89.66% | 91.03% | 396/435 | 94.71% | 412/435 | 593s | $18.72* | $4.17 | $22.89* |
Asterisks mark modeled Datalab and LlamaExtract engine costs where direct metered billing was not available in the same form as Reducto credits or native-PDF model tokens. Datalab is modeled from public processor rates: accurate convert once, segment once, and balanced extraction on routed pages per turn, before any variable balanced-extraction compute surcharge.
| Track | Engine | Schema? | Docs | Final accuracy | Median accuracy | Term recall | Terms | Engine evidence recall | Engine terms | Total latency | Median latency | Engine cost | Agent cost | Total cost | Operation pages | Cost units |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| No schema/native | Reducto schemaless | No | 10 | 95.41% | 96.15% | 97.70% | 425/435 | 98.39% | 428/435 | 152.16m | 908.2s | $41.73 | $5.41 | $47.14 | 1,391 | 2,782 |
| No schema/native | Gemini native PDF | No | 10 | 92.21% | 92.44% | 94.25% | 410/435 | 97.93% | 426/435 | 53.71m | 322.9s | $0.37 | $3.63 | $4.01 | 0 | 0 |
| No schema/native | Claude native PDF | No | 10 | 79.96% | 85.38% | 84.37% | 367/435 | 75.17% | 327/435 | 62.15m | 372.7s | $9.13 | $3.75 | $12.88 | 0 | 0 |
| Schema-assisted | Reducto schema | Yes | 10 | 94.94% | 94.61% | 97.01% | 422/435 | 99.08% | 431/435 | 169.40m | 991.5s | $41.79 | $5.10 | $46.89 | 1,393 | 2,786 |
| Schema-assisted | Datalab segment schema | Yes | 10 | 89.66% | 90.15% | 91.03% | 396/435 | 94.71% | 412/435 | 98.54m | 592.9s | $18.72* | $4.17 | $22.89* | 950 | 18,718* |
| Schema-assisted | LlamaExtract schema | Yes | 10 | 94.30% | 93.10% | 96.09% | 418/435 | 96.55% | 420/435 | 102.02m | 609.4s | $14.04* | $4.31 | $18.35* | 798 | 11,230* |
The controller asks the same user question every turn, then plans a provider query for the document engine. That query is allowed to be practical and provider-shaped, but it should not leak gold values or silently drop the broad concepts in the user turn. The audit file has 360 rows, 0 gold-value leaks, and 0 rerun candidates.
| Turn 6 provider-query audit | Track | Docs | Mean concept coverage | Min coverage | Gold leaks | Rerun candidates |
|---|---|---|---|---|---|---|
| Reducto schemaless | No schema | 10 | 99.47% | 94.74% | 0 | 0 |
| Gemini native PDF | No schema | 10 | 96.32% | 94.74% | 0 | 0 |
| Claude native PDF | No schema | 10 | 97.89% | 94.74% | 0 | 0 |
| Reducto schema | Schema | 10 | 97.89% | 94.74% | 0 | 0 |
| Datalab segment schema | Schema | 10 | 98.42% | 94.74% | 0 | 0 |
| LlamaExtract schema | Schema | 10 | 98.95% | 94.74% | 0 | 0 |
The headline remains deterministic recall. We also ran an LLM judge on the existing final answers to look for scorer blind spots. It is a diagnostic, not the leaderboard. It found no hallucinations for Reducto schemaless or Gemini native PDF, 5 for Claude native PDF, 1 for Reducto schema, 1 for LlamaExtract schema, and 8 for Datalab segment schema.
| Engine | Track | Deterministic recall | Judge recall | Judge precision | Judge hallucinations |
|---|---|---|---|---|---|
| Reducto schemaless | No schema | 97.70% | 100.00% | 100.00% | 0 |
| Gemini native PDF | No schema | 94.25% | 100.00% | 100.00% | 0 |
| Claude native PDF | No schema | 84.37% | 88.28% | 98.71% | 5 |
| Reducto schema | Schema | 97.01% | 99.77% | 99.77% | 1 |
| LlamaExtract schema | Schema | 96.09% | 99.77% | 99.77% | 1 |
| Datalab segment schema | Schema | 91.03% | 95.63% | 98.11% | 8 |
Contracts. Ten fictional but finance-realistic customer and vendor contracts for Lumen Advisory Group LLC: master service agreements, order forms, sales orders and estimates, the kind of documents an enterprise signs and later has to reason about. Each went through the six-turn review.
Invoices. Real vendor invoices from three domains (a health insurer, a telecom, a cloud provider) with every personal and company detail removed and verified clean. Plus five scanned, OCR-only invoices, and five dense multi-page bills: a cloud bill, a telecom bill, a construction pay-application (AIA G703), a hardware BOM and a spend report, generated so their math is known exactly to the cent, with subtotals nested several levels deep.
The public tables above contain the refreshed contract summary rows, provider-query audit summary and judge diagnostic summary used in sections 6-10. The invoice section is retained from the older invoice and long-document report data noted in the footer. Internal run logs and local filenames are intentionally omitted from this post.