FilingCheck
Questions about the 10-K filings of 25 public companies, answered from the filings' text and their own XBRL numbers; the model is asked to cite a passage or a fact for each claim.
The model answers with three tools. Before each tool call runs, the server checks it against the question and the data: it can correct the call or send it back to the model. Every number in the answer is then labeled against the sources the answer used.
What the measurement found
- The same rules written into the model's prompt were more accurate than the server checks. The checks are six rules the server applies before a tool runs. Written into the prompt instead, with no checks, the same rules scored 85.2% against 75.5% accuracy, at about 2.5 times the cost per question, with no clear difference on confidently wrong answers (7.3% and 10.6%).
- The pre-registered headline: against no checks, the server checks cut confidently wrong answers from 18.5% to 10.6% (95% interval of the difference -12.7 to -3.6 points).
- How it was measured: once, on 110 sealed held-out trap questions (3 runs each), by rules fixed in advance, from the tagged commit week6-measured; reproducible with make report, with no API key and no database.
What you can ask
- Numbers: the 13 metrics below, annual, for the company as a whole, from each filing's own XBRL data: revenue, net income, diluted earnings per share, operating income, pretax income, total assets, total liabilities, stockholders' equity, operating cash flow, capital expenditures, cash and equivalents, long-term debt, shares outstanding.
- Text: what the 10-K passages say, for example about risks, legal claims or currency exposure.
- It cannot answer: quarterly figures (only the annual 10-K filings are loaded); forecasts; companies outside the 25; a part of a company (a segment, a product, a region) unless the filing's text states it.
- Each question stands alone (no chat history).
The 25 companies and the fiscal years held
The list appears when the backend answers.
Examples
Try a trap question
Saved examples, chosen to show a check at work. How often the checks work is in the results.
The three labels on a number
- Verified: the figure is in a fact the answer looked up or a passage it cites, or comes from them by a calculation.
- Conflict: the figure differs from the fact its sentence cites.
- Unverified: the figure is in none of the answer's sources.
Verified means the number is in the answer's sources, not that the sentence is right.