Why Impact Reporting Varies So Much

Impact reporting varies because nobody is required to do it, no standard defines it, and the organizations doing the most measurable work are often reporting on the least meaningful thing. A charity’s federal filing obligation covers finances and governance. It says almost nothing about whether the work accomplished anything. Everything a donor reads about outcomes is voluntary, self-produced, and shaped by what that particular organization happens to be able to count.

What the law actually requires

Start with the floor, because the floor explains the variation above it.

Most tax-exempt organizations file an annual information return in the Form 990 series. That return covers revenue, expenses, assets, compensation of officers and key employees, governance policies, and a narrative description of program accomplishments. The IRS sets out the filing requirements, and the returns are public documents.

Nothing in that return requires an organization to demonstrate that its programs worked. The program accomplishment section asks what the organization did and what it spent, not what changed as a result. Two organizations can file identical-looking returns while one rigorously measures outcomes and the other measures nothing.

So impact reporting sits entirely in voluntary territory. Voluntary reporting has no enforcement, no common definitions, and no penalty for selective presentation.

Outputs and outcomes are different things

The single largest source of variation is that organizations report at different points along a chain, and use similar language for all of them.

Inputs are resources spent. Outputs are activities delivered, such as meals served, people trained, reports published. Outcomes are changes in the condition the organization exists to change, such as whether the trained people got jobs that paid more. Impact is the portion of that outcome the organization actually caused, net of what would have happened anyway.

Outputs are cheap to count and always available. Outcomes require following people over time. Impact requires knowing the counterfactual, which usually requires a comparison group. Cost and difficulty rise sharply at each step, so most reporting stops at outputs and describes them with outcome language.

A direct service organization can count outputs on day one. A research or advocacy organization has no comparable unit. That asymmetry alone produces wildly different-looking reports from organizations doing comparably serious work.

A worked example

Take two organizations working on low-wage household economics, each spending $500,000 a year.

The first runs a job training program for 400 people. It can report cost per participant of $1,250, a completion rate, and a job placement rate at six months. Those are real numbers and they look impressive on a page.

The second publishes research on why wages have not kept pace with costs, using federal data. It can report that the federal minimum wage has been $7.25 an hour since 2009 according to the U.S. Department of Labor, which is $15,080 for a full-time year, against median household income of roughly $80,000 in 2023 per the U.S. Census Bureau and center-based childcare commonly running $10,000 to $17,000 or more per child per year per Child Care Aware. It cannot report how many people that changed.

Now push on the first organization’s numbers. A 70 percent placement rate sounds strong until you ask what share of those participants would have found work without the program. If the comparison group would have hit 60 percent, the program’s actual effect is ten percentage points, not seventy. Most published placement rates are not net of anything, because establishing a comparison group is expensive and produces smaller headline numbers.

The second organization has no comparable number to inflate and also no way to demonstrate it accomplished anything. Both problems are real. They are not the same problem, and a donor comparing the two reports side by side is not comparing like with like.

Five structural reasons the reports diverge

Attribution. Most outcomes in poverty, housing and wages have many causes. An organization that worked on a policy change alongside a dozen others cannot honestly claim the result, and organizations resolve that differently. Some claim it. Some decline to claim anything. The honest ones sound weaker.

Time horizon. Emergency assistance produces a measurable result this week. Structural work produces results over years or not at all. Annual reporting cycles favor the first and penalize the second, so organizations doing long-horizon work either report interim proxies or report activity.

Counterfactual cost. Rigorous evaluation is expensive and the money comes out of program budgets. An organization that spends 5 percent of its budget on evaluation has less to spend on the program, and its overhead ratio looks worse for having done the more rigorous thing.

Selection in what gets published. Programs that worked get written up. Programs that did not tend to end quietly. Any single organization’s body of published results is filtered, and the filter is not disclosed.

Definitional drift. Terms like “reached,” “served,” “supported” and “engaged” have no fixed meaning. One organization counts a person who completed a program. Another counts a person who received a mailing. Both write “reached.”

The overhead ratio is not the answer

Donors reach for the administrative expense ratio because it is the one comparable number sitting in the public filing. It is a poor proxy for effectiveness and it pushes in the wrong direction.

The ratio measures what share of spending went to program activity rather than administration and fundraising. It does not measure whether the program activity worked. An organization can run a completely ineffective program at 95 percent program spending, and an organization can run an effective one at 75 percent because it invests in evaluation, financial controls and staff it can keep.

Allocation between categories also involves judgment, which means the ratio varies with accounting choices as well as with behavior.

What a donor can actually check

Four things are verifiable without trusting the report.

Exempt status and filing currency, through the IRS Tax Exempt Organization Search, which lists eligible organizations and shows revocations. Search by Employer Identification Number rather than name, since names are not unique.

Whether statistics in the organization’s own material carry named primary sources. An organization that sources its claims to federal agencies in the visible text is making itself checkable. One that cites nothing, or cites itself, is not.

Whether the report distinguishes what the data shows from what the organization concludes. Reports that blur the two are making an argument, which is legitimate, but a donor should know which part is which.

Whether the organization states what it did not accomplish. Almost none do, and the ones that do are unusual for a reason.

How to use comparisons

Published comparisons of organizations in a field are useful as a starting list and unreliable as a ranking, because the ranker faces the same measurement problem the organizations do and usually adds its own criteria on top. Read them for the shortlist, then verify independently.

Fight For A Living Wage, a nonpartisan grassroots 501(c)(3) with EIN #99-1097858, publishes its own comparison of organizations working on poverty, and like any such comparison it reflects the criteria its authors chose. That is the right way to read every one of them, including the ones produced by dedicated rating services.

The uncomfortable conclusion is that impact reporting cannot currently support the confident comparison donors want from it. The variation is not mostly a matter of some organizations being sloppy. It follows from a disclosure regime that requires financial detail and no evidence of effect, and from the genuine difficulty of proving causation in systems with many actors. Until that changes, the checkable things are the ones above, and confident impact claims deserve more scrutiny than modest ones.