Quick benchmark today comparing Astra 6, Fable 5.1 and Opus 5:
I have a build plan in my project which has step by step MDs to get to the next milestone. I had GLM 5.3 build 3 of the steps that could be built in parallel. The benchmark was for these 3 models to review the code. They were told to review the code for those 3 commits, not the whole project.
Opus 5 was run with Claude code on the $20 plan, the other 2 were used through opencode with API billing.
I had all of them review the changes with the same prompt, then I gave them all the outputs from the other 2 agents and asked them to rank themselves and the other 2.
Here is the results
Astra ranking:
A, O, F
Opus ranking:
O/A tie, F
Fable ranking:
O, A, F
They were not told the names of the other agents they just knew which ones were their own. All 3 reported fable as the worst because it missed bugs that the others caught.
Opus and astra essentially had the same number of severe bugs found, but the most bugs any one found was 13 while the 3 had 23 unique bugs found combined so it still pays to have multiple models review your code, at least at important milestones. They really do catch things that others miss.
As for cost, astra and fable cost the same per token, but astra spent $6.67 while fable cost $7.48. Opus used up 50% of a 5hr limit on the $20 plan. Opus's overperformance might be a result of the Claude code harness, but fable and astra were both on opencode and fable did worse there while spending more. Also, those numbers are for both reviewing, and comparing the reviews, which included testing whether the findings were actual bugs or false positives. Astra was like $2.50 compared to $5.50 on Fable on just the reviewing step so it really outperformed fable at half the token usage. Opus also found one of its own bugs to be a false positive. Astra and fable found no false positives in any of the agents findings.
Me personally, I will probably use astra when I need something smarter than opus, but my daily driver will still be some cheap Chinese model for implementation and opus for reviewing since I have the sub.
I have a build plan in my project which has step by step MDs to get to the next milestone. I had GLM 5.3 build 3 of the steps that could be built in parallel. The benchmark was for these 3 models to review the code. They were told to review the code for those 3 commits, not the whole project.
Opus 5 was run with Claude code on the $20 plan, the other 2 were used through opencode with API billing.
I had all of them review the changes with the same prompt, then I gave them all the outputs from the other 2 agents and asked them to rank themselves and the other 2.
Here is the results
Astra ranking:
A, O, F
Opus ranking:
O/A tie, F
Fable ranking:
O, A, F
They were not told the names of the other agents they just knew which ones were their own. All 3 reported fable as the worst because it missed bugs that the others caught.
Opus and astra essentially had the same number of severe bugs found, but the most bugs any one found was 13 while the 3 had 23 unique bugs found combined so it still pays to have multiple models review your code, at least at important milestones. They really do catch things that others miss.
As for cost, astra and fable cost the same per token, but astra spent $6.67 while fable cost $7.48. Opus used up 50% of a 5hr limit on the $20 plan. Opus's overperformance might be a result of the Claude code harness, but fable and astra were both on opencode and fable did worse there while spending more. Also, those numbers are for both reviewing, and comparing the reviews, which included testing whether the findings were actual bugs or false positives. Astra was like $2.50 compared to $5.50 on Fable on just the reviewing step so it really outperformed fable at half the token usage. Opus also found one of its own bugs to be a false positive. Astra and fable found no false positives in any of the agents findings.
Me personally, I will probably use astra when I need something smarter than opus, but my daily driver will still be some cheap Chinese model for implementation and opus for reviewing since I have the sub.
1
