GPT-6 Astra: evaluate the whole task

The interesting test for a new coding model is a small, awkward feature in an existing repository. Give it an incomplete bug report, a real constraint, and a way to check the result. See whether it can bring those pieces together without someone steering every step.

That is the test I would start with for GPT-6 Astra. OpenAI describes it as its most capable model for complex work across coding, research, computer use, and documents. Its model card lists a 1,050,000-token context window, image input, function calling, and structured outputs. Those are documented capabilities, rather than results from a benchmark I have run. Official GPT-6 Astra model documentation.

Model details in this article were checked on September 5, 2026.

Start with a task that has a finish line

Imagine a booking application where a customer can submit the same request twice by pressing the button again during a slow response. A useful assignment would include the symptom, the expected behavior, and the constraints around existing bookings.

The finish line should describe observable behavior: repeated submissions with the same request identifier create one booking; independent requests still work; the interface shows a useful pending state; failures remain recoverable.

This forces the evaluation to span the browser, API, persistence layer, and verification. A model can produce a convincing frontend patch while leaving the actual duplicate-write problem intact. The test needs to catch that.

I would keep the assignment small enough that a human reviewer can independently establish whether the result is correct. Otherwise, the evaluation quietly becomes a test of how persuasive the final explanation sounds.

Give context a structure

A large context window provides room for more material. It does not decide which material is authoritative.

For this booking task, I would provide the repository instructions, the relevant schema, an example request, and the acceptance criteria. The agent could inspect adjacent code as it develops a hypothesis. A folder full of historical design notes should not carry the same authority as the current database constraint.

The handoff should also distinguish facts from guesses. “The API returned two booking IDs” is evidence. “The button probably fired twice” is a hypothesis. Keeping that distinction visible makes it easier to notice when the agent has fixed an explanation instead of the problem.

Score the work in separate dimensions

A single pass/fail score hides useful differences. I would record these outcomes separately:

1. Diagnosis: did it identify the path that permits duplicate writes?
2. Implementation: does the patch enforce the intended behavior under concurrent requests?
3. Verification: did it exercise both the duplicate case and a normal submission?
4. Scope: did it preserve unrelated behavior and explain any migration requirement?
5. Handoff: can another engineer understand the change and its remaining limits?

The distinction matters when comparing models. One might diagnose correctly but stop before checking the browser. Another might complete more steps but introduce an unnecessary rewrite. Those failures need different interventions.

Keep authority in the application

More capable planning does not change who is allowed to perform an action. The runtime should still enforce workspace boundaries, tool permissions, and the point where a deployment needs approval.

For the example task, editing an isolated checkout and running local checks can be part of the assignment. Applying a production migration is a separate operation with different consequences. That boundary should exist in code and permissions, even when the model's explanation is excellent.

This is the same engineering concern behind my notes on prompt, context, loop, and graph design: the application has to make its control flow understandable.

Compare completed outcomes

I would compare Astra with the current model on the same tasks, with the same tools and acceptance criteria. Record elapsed time, total usage, retries, and reviewer corrections. Repeat ambiguous cases before drawing conclusions.

My adoption criterion would be a measurable improvement in accepted work: more correct tasks completed, or less human correction at an acceptable cost. A compelling demo earns a place in that evaluation. The evaluation earns a place in the product.

For the backend problem used in this example, see designing APIs that are safe to retry.