Dia: Six Months of Real-World AI Use

Is Dia actually good for everyday AI work?

Yes.

For research, discovery and everyday brainstorming, Dia was useful.

Getting started was almost frictionless. I could point it at an open tab, give it a URL or paste in a brief and get structured, specific work back without much setup. That matters when you're working through ideas. The distance between a question and something useful coming back was small.

On 13 April, my first day with it, I asked Dia to assess its fit for my workflow. It described memory across sessions as a feature and was candid about the role I would still need to play:

That turned out to be an important part of the experience. Because the further I pushed Dia into actual work, the more obvious that quality gate became.

Does Dia remember what happened last week?

This became one of the biggest problems. Within a long chat thread, Dia could hold context well for about a week. But nothing carried between separate sessions.

1. April

   In April, Dia had described memory across sessions as part of its fit for my workflow.

2. July

   By July, Dia said plainly that nothing carried between sessions.

The practical consequence was bigger than simply having to repeat myself. I ended up maintaining a Notion stack containing my Master Context, Session Primer and AI Handoff documents purely to give Dia the context it had lost.

The AI was useful. But I was doing work to make the AI useful.

That distinction matters.

What happens when an AI can't access the work?

This was where the ceiling became much more obvious. I was working on a hands-on Eleventy website. Dia could advise me about the work. It couldn't actually get into the project and work on it.

So I became the relay. I had to move information between the environment where the work was happening and the environment where Dia could reason about it.

At one point, showing Dia a render meant taking a screenshot, resizing it with sips, and dropping the resulting file into its work folder.

None of that makes Dia a bad browser. But from my point of view, the result was the same:

I had become part of the integration layer.

Can Dia actually see the work it's helping with?

This became particularly obvious when the work turned visual. Dia could discuss design. It could reason about content and structure. But it couldn't see the visual styling in the way I needed it to. Image attachments reached it as text descriptions, not pixels.

As Dia itself put it on 12 May:

On a project where visual design mattered, that meant the final judgement still came back to me. I had to look at the thing myself.

Again, the issue wasn't that Dia had no useful design input. It did. The issue was understanding the difference between reasoning about something and actually having access to the thing you're asking it to judge.

Can Dia actually take action?

For the work I was doing, Dia was largely read-only. It could advise. It could analyse. It could help me think through a problem. But it couldn't simply go and make the required change.

As it put it on 7 May:

Usage ceilings also cut some passes short mid-edit. That meant build work needed another tool. Eventually, that meant using Cursor and later Claude Code for work inside the repository.

This created a recurring pattern:

Why did Dia become frustrating to use?

Two things came up repeatedly.

Those two things — reliability and response time — became much more important over six months than they were during the initial impression.

Where did Dia genuinely save me time?

Despite all of that, Dia was not a failure. Some of the most useful evidence came from the things it did well.

After a holiday break on 4 September, Dia was able to rebuild the full project state from my open tabs. When I challenged a detail it had wrong, it corrected itself.

It mapped 774 commits across five branches and eleven tags to help identify a restore point from 19 June.

On 9 May, it also pushed back on another tool's uncritically positive feedback rather than simply agreeing with it. And even though it couldn't see the visual design directly, parts of its design critique remained useful.

That matters.

How far can an AI tool take you before you have to take over?

This became one of the most useful questions I took from the experience. In the retrospective, I called this the Operator Ceiling.

How far can the tool take the human operator before the human has to take over?

For Dia, the answer was fairly far.

That handoff is part of the real-world experience of using an AI tool, whether the product itself considers it part of the product or not.

What happens when one AI tool isn't enough?

This is where the six months changed how I think about AI tools. It is easy to evaluate a tool in isolation. But I don't work in isolation.

I work in workflows.

Dia was useful. Cursor was useful. Claude Code was useful. But usefulness doesn't automatically add up to a good workflow. Sometimes it creates a chain of handoffs. And the more handoffs there are, the more I have to manage the system around the AI.

That creates a hidden cost. The tool might save me time on one part of the job while creating additional work somewhere else.

How much work did Dia create around itself?

This is probably the most important distinction I took from the experience. An AI tool doesn't have to be completely useless to fail my workflow. It can be genuinely useful and still create too much work around itself.

None of those things appeared particularly significant at the start. Over six months, they accumulated.

That is why I think workflow integration matters as much as the individual quality of an AI's answers.

Would I actually pay for Dia?

After six months of real use:

No.

That doesn't mean I got no value from it. I did. It means the value wasn't enough at the level I needed.

The strongest scores were around getting started and the depth of the core research/discovery use case. Reliability and output quality pulled the overall result down.

And although I got real value from Dia, I didn't get enough value to pay for it at the standard my work required.

What is Dia actually good for?

That's more useful to me than simply saying Dia is a 63/100 product. It tells me where I'd actually use it.

Does an AI tool have to be bad to fail a crash test?

No.

This is probably the most important conclusion. Dia wasn't bad. It was useful. It just wasn't useful enough for everything I wanted it to do.

A crash test shouldn't be designed to prove that a tool is bad. The useful question is:

Where does it stop being good enough for the person relying on it?

• Sometimes that will be an obvious failure.
• Sometimes it will be a hallucination.
• Sometimes it will be a missing capability.
• Sometimes it will be slow enough to break the workflow.
• Sometimes it will simply be the point where the human has to take over.

That boundary is valuable evidence.

What did six months of using Dia teach me about testing AI?

This experience happened before the CTAI Test Rig existed. It wasn't a controlled experiment. I didn't log every hallucination. I didn't run the same prompts every day. I wasn't following a fixed test script. I was doing my work.

And that is precisely why I don't want to dismiss the experience because it happened before the Rig. It tells me something a controlled test doesn't always capture. It tells me what happened when an AI tool became part of my actual workflow.

But it also taught me that not every test needs to last six months. The value came from the real work, not simply from the length of the calendar.

The duration should follow the question. Not the other way around.

Where does the human still have to take over?

This is why I'm building CTAI around a human verdict. Dia itself told me early on that I needed to remain the quality gate. I agree.

AI can be useful equipment. It can help execute. It can help analyse. It can help find patterns. It can help prepare the next test.

But when the question is:

Would I actually trust this with my work?

the answer still needs to come from the person doing the work. That is the difference between an AI tool producing an answer and an AI tool being good enough to depend on.

What did Dia change about the way I think about Crash Test AI?

The biggest lesson wasn't actually about Dia. It was about the kind of testing I want CTAI to do.

That's where the interesting evidence appears.

1. Does it remember?
2. Can it access the work?
3. Can it see what you need it to see?
4. Can it take the action?
5. Does it hallucinate?
6. Does it slow you down?
7. Where does another tool become necessary?
8. Where does the human have to take over?

And ultimately:

Can it produce work I'd actually trust enough to use?

Those are much more interesting questions to me than a feature checklist.

My Dia verdict

After six months of daily use, Dia was genuinely useful for research, discovery and brainstorming. It helped me think, explore, reconstruct context and sometimes challenge assumptions.

But it also hallucinated too often, became frustratingly slow, couldn't access the build environment, couldn't carry context between separate sessions, couldn't see visual work in the way I needed, and couldn't take the actions required to get the work over the line.

For my workflow, that created too many handoffs.

I'd use Dia again for the things it does well. I wouldn't make it the only tool responsible for getting the work over the line.

And that's probably the most useful conclusion a crash test can give you:

Not whether a tool is good or bad. Where it is good enough — and where it isn't.

Test disclosure

The crash-test starts where the demo ends