The Big AI Secret · ongoing research

AI's return is a feeling. Sometimes, that feeling is doubt.

Agency call accounts reveal a gap between using AI and knowing what it contributes. They also show why a useful measurement must distinguish productivity, quality and benefits that nobody has a baseline for.

26 October 2026 · 4 minute read

The revealing part is not simply that an agency has no measured return. It is that the uncertainty runs through the team's experience of the work. People are using AI, yet the speaker reports doubts about whether it makes the work quicker. That is a different finding from enthusiasm without a spreadsheet. The benefit itself remains unsettled.

“It's a feeling, and I will add that it's a feeling that multiple people have kind of started saying that sometimes we don't know if it's saving us time or not.”Agency and marketing, under 25 staff

This week's notebook examines measurement within agency call extracts for The Big AI Secret. These are accounts of work, not audited productivity records. They let us inspect how people describe value and where their evidence stops. They do not establish how common these experiences are across agencies. Roles are not recorded for the quoted speakers, so the attributions leave that gap visible.

Use is visible. Value is harder to pin down.

A person can know they use AI regularly without knowing whether it improves the completed task. Generating something is visible. Establishing whether the whole job became easier requires a comparison. It would be tempting to blame checking, rewriting or poor outputs, but the extracts do not establish any of those explanations.

The answers identify a measurement gap, not a failed use of AI. Without formal tracking, useful work may go unrecognised. Equally, a task that feels faster may not improve productivity once the finished result is considered. The evidence supports neither conclusion here. What it supports is a narrower statement: the speaker cannot point to a formal record of the benefit.

Sometimes the baseline predates the person

“It was in place when I joined, so I can't speak beforehand, but it would save me hours to manually pull the reports, to schedule, to go in and post every day.”Agency and marketing, staff size not recorded

Here the missing comparison has a specific cause. The speaker arrived after the system was installed. They can describe the manual work it replaces, but they did not experience the earlier process. Their account is useful evidence about what the system does. It is not an observed before-and-after comparison of their own work.

This distinction matters as AI becomes ordinary workplace equipment. A new starter may inherit reporting or publishing processes without knowing how they were previously handled. Asking that person for a return figure risks producing an imagined comparison rather than a recorded one. The missing evidence may sit in old work records, or it may never have been collected.

The practical implication is to preserve the baseline when a process changes. Record what the task involved, what counted as finished and what people had to check. If that record does not exist, say so. A retrospective estimate can still inform a decision, but it should remain labelled as an estimate rather than quietly becoming a measured result.

Quality needs its own evidence

“I'm not sure if it saves time, it increases quality.”Agency and marketing, under 25 staff

This speaker described AI used for quality assurance. In their account, team members see flags in a shared spreadsheet, verify the output and make changes themselves. The claimed benefit is better quality, not confirmed speed. Treating the absence of a productivity figure as proof of failure would miss what this use is intended to contribute.

Yet quality should not become a refuge from measurement either. The call gives a claim of improvement, not a documented quality assessment. A suitable follow-up would examine whether flags identify genuine problems, whether suggested changes are accepted and whether the finished work needs less correction. Those are proposed checks, not outcomes established by these extracts.

Make the claim match the record

Taken together, these accounts describe different evidence problems. Doubt about productivity needs observation of the task. An inherited process needs a credible baseline, where one can be recovered. A quality claim needs a quality test. Folding them into a general question about whether AI is working would blur distinctions that matter to a director deciding what to keep.

The next useful step is not a grand return claim. It is a clear statement of the expected benefit, attached to a record that could confirm or challenge it. These calls do not show that agency AI lacks value. They show why confidence in using it and evidence of its value must be kept separate.