Evaluating AI Agents as Products

Coding benchmarks evaluate AI agents on well specified tasks. Open ended qualitative checks show which can make the nicest demo but leave enough flexibility that they don’t measure the rigidity of real work. Neither capture human-in-the-loop performance nor do they speak to efficiency and productivity. To explore how we can better measure how “useful” AI agents are, I introduce three tasks to generate data labeling interfaces in an (ideally) short interactive coding session. The specified interface functionality combines mandatory requirements with open ended choices, allowing us to evaluate efficiency, collaboration, and taste, providing a proxy for how well the agents perform as tools in real coding situations. Here I introduce the tasks and evaluate five agents – three smaller local models and two near frontier.

When we started doing ML, benchmarks scores pretty closely captured the performance of a model. If a classifier had a certain precision and recall, as long as the data distribution didn’t change this told you what you needed to know. I think AI’s origins in simpler ML models, and the current narrow gap between academic advances and commercial offerings has kept an outsized emphasis on benchmarks.

Benchmarks can be most useful when (a) we want to compare models and (b) we are in a period of rapid model advancement. And in the brief history of LLMs, model performance was probably a suitable proxy for product performance. Anything you build on GPT-5.6 is going to be better than if you’d built it in GPT-4. A known issue of course if that benchmarks saturate and all models can basically “ace” old benchmarks making them useless for comparison. Less discussed is that the entire concept of model benchmarking may be saturating in many applications, and the remaining alpha is in more product-like qualities around user experience and the elusive “taste”.

The Artificial Intelligence Coding Agent Index¹ is a composite index of three public benchmark datasets (DeepSWE, Terminal-Bench V2, SWE-Atlas-QnA). The first two of these have agents do software engineering tasks with well defined expected behavior. The last one has the agent answer questions about a codebase.

I would contend that such evaluations no longer align with how effective a given tool (say Cursor or Codex or Claude Code) is at improving developer outcomes (saving time, shipping more, etc). Well specified tasks test for a different set of behaviors than more open-ended real-world problems where taste and judgment are necessary². Furthermore, we’re at a point where most agents can basically solve any task when a human is in the loop. Finding tasks they can’t solve becomes increasingly about tricking them or identifying edge cases instead of testing which is more useful in the real world. The relevant question is no longer “can it do this” like it was in the early days, but how efficiently does it do it (with a human in the loop), does it exercise good judgement that leads to a quality product, does it provide helpful input as a co-worker, etc.

On the other hand, there is a danger in open-ended testing without clear goals. Recently Andrej Karpathy suggested prompting for video generation³, and there are analogous coding focused tasks like one-shotting video games. These make cool demos but are too open ended to realistically test utility. AI’s perennial problem is that it’s great for demos but real world considerations take the wind out of its sails. A Netlify blog post also compared how well models could single-shot web design tasks⁴. This has a similar problem, it doesn’t test against any subjective use criteria, and it only measures single-shot performance, not interaction, so doesn’t obviously predict how useful a tool will be when used with a human in the loop.

To explore how useful different coding agents (consisting of a model and harness) are, I wanted a set of tasks that was specific enough to have constraints and necessary success criteria, while also involving sufficient flexibility to let models demonstrate their taste. Crucially I also wanted real useability criteria. I didn’t want to just build something and say it passed the tests and looked well designed, I wanted there to be the equivalent of a user that could decide if the end result met their needs. The goal is not to fully simulate real use, but to find proxy tasks that give a sense of how efficient and effective the tools are in real situations by simulating meaningful interaction and success criteria.

Based on these criteria I experimented with tasks requiring agents to build software in support of dataset labeling or annotation. This is a task I use AI for frequently in real life. It involves clear functional criteria – a user is presented with a set of data points and needs to make some determination(s) about each (add a label), so the software must support that. It involves taste and judgement in terms of how the material is presented, the labeling flow, how progress is saved, etc. And its ultimate success rests in how effectively the user is able to use the software to perform the labeling.

Tasks

General criteria: some simple tasks requiring human judgement that could be completed rapidly. Sufficient volume to get a feel for how efficiently a user can run through many samples. Meaningful upside in how the information could be laid out to support the user in their task.

Test Setup: The models tested so far are Z.ai GLM 5.2, Qwen 3.6 35B A3B (8-bit quant), Laguna Poolside 2.1 (4-bit quant), Meta Muse Glimmer (8-bit quant), and Claude Opus 5. Claude was used with Claude Code and the rest were used with stock Pi (another coding harness). I used whatever the default thinking was for each and this isn’t meant to be a detailed comparison between thinking modes. The models were selected based on ones I wanted to use for one reason or another. Claude because it’s generally good and I’ve used it for a long time, GLM as a modern open weights model, and the three smaller ones as examples I can run locally with different tradeoffs between active and total parameters.

Test Procedure: For each of the three tasks I used GLM to build a reference implementation in an interactive session beginning with the prompts mentioned earlier. I used each of these sessions to decide on an appropriate end-state that captured the base functionality I wanted to see in the application. I approached the task as I would in a real development setting, iteratively running the application, identifying issues or refinements, and them prompting to address them.

For the subsequent agents I replayed, to the extent possible, the sessions, beginning with the same starting prompt, and using the same follow-ups when applicable, but adjusting as necessary for different issues that arose.

Evaluation: We can look at the number of tokens used and the number of user turns required as measures of efficiency and native taste. Fewer turns indicates the model arrives at what I consider a good result faster, despite the underspecification of the initial prompt. The more interesting part of the evaluation comes from actually experiencing the interaction and the output and is mostly manual and qualitative. This may be unappealing in comparison with automated benchmarks, or feel like a cop-out. The goal is to replicate “product testing” however, which in other domains, say running shoes or cars, ultimately boils down to using something and seeing how well it works for you. One quantitative measure that this method supports but was not attempted is actually timing how efficiently and accurately the downstream data labeling task can be completed using the different tools. This, combined with the development time and manual steering required, could give a firmer quantitative picture of how effective the different agents are.

In the qualitative evaluation we look for what stylistic choices were made to support the tasks, how much intuition or common sense the agent had about the task, and the apparent ease of use of the developed solution. We also consider the overall experience and impression using the agent. In running the evals, they felt very representative of the kind of interactions I have with coding agents, including the debugging side quests and need for steering.

Quantitatively, we can also measure how many turns were taken and how many tokens were used, both proxies for efficiency.

Apps

Z.ai GLM 5.2 / Pi

On the class labeling task, GLM needed to be told to add a sidebar, took too long thinking and reached the default token limit twice, and initially messed up the syntax highlighting, breaking the layout whenever it highlighted a word. It particularly struggled when asked to come up with a way of highlighting words and waffled on trying to make a word list. It also made (in my view) a poor design choice in picking red and green for the two class colors, given the good/bad connotation of these two colors.

For the ranking task, GLM made near black-on-black interface to draw the pictures at first which made it impossible to see. It also displayed a bunch of metadata that had no relevance to the task.

On the editing task, GLM had some initial scroll and layout issues, and displayed extraneous information – the instructions specify that this is a test using just the python error message but it displayed other info anyway.

Overall GLM generated easy to use applications and made good stylistic choices. For example, it up front made the sentence large and easy to read for the class labeling task. For the ranking task, there was a but if busyness in the interface and the dark mode made it hard to read, though I retrospectively noticed it added a light / dark toggle. The editing interface requires scrolling to see the whole main panel which makes it slower to use.

Qwen 3.6 35B A3B / Pi

Qwen didn’t make a great use of space in the class labeling app, the sentence to be reviewed was smaller than it needed to be. Similar to GLM, Qwen didn’t initially add a scroll bar, and messed it up on its first pass. It also had trouble with the layout when applying syntax highlighting.

On the ranking task, Qwen didn’t fully finish according to what I wanted but completed something that worked. It ran into errors that needed multiple human-in-the-loop debugging rounds, first with loading the pictures and then dragging and dropping. It also initially displayed the pictures in black-on-black.

For the editing task, Qwen initially shared information it was not supposed to with the user, required steering on the error highlighting, had an error in the button states that needed manual correction

Laguna Poolside 2.1 / Pi

For the class labeling task, Laguna initially built a terminal application. It did however respond to the prompt to create a web-app but adding a sidebar automatically. It didn’t display the full sentences initially which is poor judgement in an application specifically for reviewing sentences. Laguna also picked red for one of the class colors.

For the ranking task, Laguna made only minor judgement errors such as displaying the images in a stack that needed scrolling at first.

Laguna experienced a number of problems on the annotation task: scroll and layout issues, timeout due to excessive thinking, errors highlighting error location and with persistence, and an issue with loading an external library (pyodide) that required some back and forth.

Meta Muse Glimmer / Pi

Muse also generated a terminal app to start with for the class labeling task and didn’t initially add a side scroll bar. It initially didn’t show how each sample had been labeled and only that it had been. It performed well on the syntax highlighting in that it pursued a quick, simple solution. GLM, Qwen, and Laguna each took an outsized amount of thought to come up with highlighting schemes.

For the ranking task, Muse took additional back and forth to resolve some minor bugs that surfaced about display and dragging.

On the annotation task, Muse didn’t initially add a sidebar, some loading issues, left in error information it shouldn’t have, and popped up an alert between evals which slows the user down.

Claude Opus 5 / Claude Code

On the class labeling task, Claude also didn’t add a scroll bar at first, otherwise it correctly built the app with minimal steering.

Claude mostly single-shotted the ranking task, the only issue was that it focused on speed only (clicking the pictures in order to rank) but didn’t include the fallback cases of moving them around from the outset.

For the annotation task, Claude added extra information that should have been hidden from the user, and lacked judgement about sidebar labels, labelling by error type when they were substantially all the same error type.

Efficiency

The tables below compare token counts and turns for the different tasks. Note that Claude reports its token use slightly differently and most models don’t differentiate between output and reasoning tokens. Muse consistently uses a smaller number of tokens than the others, while taking about an average number of turns. Claude is always the fewest turns, largely because it includes lots of internal testing and debugging within turns. While it doesn’t appear overly token-heavy, the usage here apparently cost ~$50. Both Qwen and Laguna had instances where the interaction took an outsized number of turns due to some bug that required iteration.

Class Labeling

Agent	Input	Output	Reasoning	Cache read	Cache creation	Total	Turns
GLM 5.2 / Pi	89,139	127,572	106,284	1,403,456	—	1,620,167	6
Qwen 3.6 35B A3B / Pi	107,722	33,518	0	2,204,257	—	2,345,497	9
Qwen 3.6 35B A3B / Pi	162,784	34,994	0	2,535,413	—	2,733,191	9
Laguna 2.1 / Pi	90,829	109,040	0	4,253,740	—	4,453,609	5
Muse Glimmer / Pi	16,405	19,201	0	663,177	—	698,783	5
Opus 5.0 / Claude Code	281	155,820	—	11,022,272	165,744	—	4

Ranking

Agent	Input	Output	Reasoning	Cache read	Cache creation	Total	Turns
GLM 5.2 / Pi	116,016	84,130	48,363	2,562,304	—	2,762,450	4
Qwen 3.6 35B A3B / Pi	141,296	75,554	0	5,533,304	—	5,750,154	14
Laguna 2.1 / Pi	70,904	88,254	0	6,716,033	—	6,875,191	5
Muse Glimmer / Pi	25,637	34,620	0	4,809,671	—	4,869,928	10
Opus 5.0 / Claude Code	199	166,573	—	7,528,675	697,690	—	2

Editing

Agent	Input	Output	Reasoning	Cache read	Cache creation	Total	Turns
GLM 5.2 / Pi	97,851	89,789	53,014	4,403,648	—	4,591,288	6
Qwen 3.6 35B A3B / Pi	108,892	38,328	0	4,520,312	—	4,667,532	9
Laguna 2.1 / Pi	130,250	291,206	0	7,919,440	—	8,340,896	18
Muse Glimmer / Pi	34,558	28,456	0	5,200,109	—	5,263,123	8
Opus 5.0 / Claude Code	284	190,226	—	13,309,141	626,046	—	3

Overall Impressions

Claude + Claude code is clearly the “smartest” and most capable of the models, which should be no surprise and shows in its ability to nearly single shot many of the tasks, with only minor tweaks necessary, and its generally good judgement. What I like the least about it is what I call “co-working” – it does way too much useless stuff, particularly extensive testing, without asking. This makes any iteration that does occur take way longer, and of course uses more tokens. Some people might appreciate the rigor, I found it overkill for these tasks and that it made it hard to collaborate with. I would have preferred a few extra rounds of back and forth than sitting and waiting for it to run yet another test of something. It’s important to note that Claude Code itself probably plays a big role here, I did not try Claude with Pi and this may significantly reduce the testing it does.

Note added in draft: While I didn't use any AI to write this, proofread it, etc., I did get Claude Code to apply some formatting to the HTML version. Ironically it again spent an outsized amount of time running various tests, even when it had clearly read what I wrote above, I felt at some level like it was rubbing it in.

GLM , for all intents and purposes is as capable as Claude for the tasks tested here. Claude doesn’t share it’s reasoning tokens so we can’t compare it, but GLM felt very verbose, like it waffled a lot before coming to decisions. In one case it timed out because it just went on thinking instead of ever concluding. My sense is also that it’s judgement might be subtly worse than Claude’s – it needed a little more steering. Overall it was easy to work with (in Pi) and I appreciated not having to sit through any excessive validation. The end result didn’t seem any worse for it.

I will lump Qwen and Laguna together. These are capable small models (Laguna is bigger but had to be quantized to fit on my machine), but the gap with the frontier models shows on a task like this. Both took consistently more steering to get the apps right, and lacked some of the judgement the bigger models could exercise. Ultimately the interfaces they generated were simpler looking and had subtly lower production values, however for an application like this it doesn’t really matter. I found the autonomy of the models good – they did their thing but involved me as necessary didn’t go off in any side-quests of their own (*cough* Claude) and supported me well in debugging. They both also think too much and get stuck in long or infinite thinking loops sometimes, failing to make a decision.

Muse came out as I was running the tests and I decided to include it to see how it differed from the other two small models. It’s particularly unique in how it thinks. It’s extremely terse and to the point, has a completely different style (no “Wait, actually…” like all the other ones, and generally is very action oriented. This is a dense model so is much slower running on my machine, but the overall interactions ended up faster than with Qwen or Laguna because of how direct it was. I had expected speed to be the biggest barrier to using it. In terms of performance, it was as good as or better than the other two small models in terms of judgment and the steering it required.

A summary of overall impressions is below. It's subjective but based on my experience building the different apps and the observations above.

Conclusions

Appendix

Below are the user turns for the different sessions. While I didn’t include the agent turns in the interest of length, the user turns give a sense of the interaction.

Class Labeling

Z.ai GLM 5.2 / Pi

Agent	Taste	Co-working	Intelligence	Verbosity
GLM 5.2 / Pi	◕	●	●	◕
Qwen 3.6 35B A3B / Pi	◑	◕	◑	◑
Laguna 2.1 / Pi	◑	◕	◑	◔
Muse Glimmer / Pi	◕	●	◕	●
Opus 5.0 / Claude Code	●	◑	●	◕

## user

In this directory is a file that contains random sentences from wikipedia articles about either the 2008 financial crisis or the dotcom bubble. I want to make an application that displays this data for a person to review and manually label as one or the other. The data is labeled but ignore those labels. Our goal is to allow the user to review and decide as quickly as possible the article from which each sentence originates

## user

Can you add a panel on the left that contains a scrollable list of all the sentences so we can rapidly move between them?

## user

Great. Can you add some heuristic syntax highlighting to make it easier for the user to tell at a glance some of the features that might predict which article the sentence came from? Don't bother doing any testing of how well they perform (since we don't have a train test split that would be cheating), let's just experiment with some basic highlighting. For example anything that's a year would be telling, words related to finance or mortgages vs words related to technology would be telling. Before doing that, please change the coloring away from red/green (which to me implies good / bad) to something more neutral but still with good contrast

## user

You errored due to max token limit, keep going please

## user

OK there have been two passes where you reached to max token limit overthinking the heuristics to use. I'm just looking for a simple set of words or terms to highlight, it doesn't have to be exhaustive or 100% correct, it should just be a net help to the user. Err on the side of simplicity and come up with some basic highlighting rules without overthinking

## user

The highlighted words are in their own divs or something that don't flow with the rest of the sentence and the text wraps within them, so instead of just appearing as highlighted words they completely change the layout of the text and make it unreadable

Qwen 3.6 35B A3B / Pi

## user

Can you add a panel on the left that contains a scrollable list of all the sentences so we can rapidly move between them?

## user

It seems like I can't actually navigate with the side bar, when I click on a sentence it doesn't go there, it just advances to the next one as I grade them

## user

There's still something wrong. The text wraps separately in different sections. It's like there is only ever one row that gets divided into columns, the highlighted words go in a column, and then the interstitial non-highlighted text goes in adjacent columns and wraps when its too long to fit the column which is always

## user

Great that worked. Now is there a way to export the data as a json once it's labeled?

## user

Can you make it so I can download it at any intermediate stage and not just when done?

## user

It looks like I'm stuck on the done screen now when I load the app and not able to label any

Laguna Poolside 2.1 / Pi

## user

OK that's a good start - I want to be able to move back and forth more easily though and not just step through. Can we make this into a web app to give some more flexibility?

## user

OK looking good. I noticed it doens't display the full sentences on the web app, they get truncated after a certain length. Can you fix that please?

## user

Great. Can you add some heuristic syntax highlighting to make it easier for the user to tell at a glance some of the features that might predict which article the sentence came from? Don't bother doing any testing of how well they perform (since we don't have a train test split that would be cheating), let's just experiment with some basic highlighting. For example anything that's a year would be telling, words related to finance or mortgages vs words related to technology would be telling. Before doing that, please change the coloring away from using red (which to me implies bad) to something more neutral but still with good contrast

Meta Muse Glimmer / Pi

## user

OK that's a good start - I want to be able to move back and forth more easily though and not just step through. Can we make this into a web app to give some more flexibility?

## user

Can you add a panel on the left that contains a scrollable list of all the sentences so we can rapidly move between them? Also I'd like to be able to see which way a sample was classified when I go back to it, right now it looks like they all just get highlighted in yellow to indicate they have been labeled but I don't see the actual label anywhere

## user

The operation stopped for some reason, I may have hit escape by accident

Claude Opus 5 / Claude Code

## user

Can you add a panel on the left that contains a scrollable list of all the sentences so we can rapidly move between them?

## user

Is there a way to export the current progress back to a json? I didn't see anything

Ranking

Z.ai GLM 5.2 / Pi

## user

in ./output are 50 folders each corresponding to a symbol and containing five attempts by an LLM at drawing that symbol as a svg. Can you make me a ranking interface that lets me view the five for each symbol and rank them from best to worst. Don't overfit anything to the data, it should work for other shapes in this format as well.

## user

I can barely see the shapes, it looks like they are drawn in a thin black line on a black and dark blue checkered background, can you make them thicker and add contrast please

## user

The metadata about each drawing isn't really important to assessing quality, and right now the UI is a bit clumsy. I want to be able to sort them as fast as possible, can we modify it so the initial images are displayed and can be dragged into a row of slots below that correspond to their rank order?

## user

Looks good, one last minor thing, can you make clicking on an image in the top row default to moving it to the next available rank, so if the user wanted to they could just click them all in rank order (best to worst) and it would complete the task without needing to drag. They should still be able to move them around as before if they want

Qwen 3.6 35B A3B / Pi

## user

It runs but all I ever see is "loading" - is it supposed to take a really long time?

## user

The spinner spins forever, there is an uncaught syntax error, invalid assignment left-hand side one line 591 of rank.html according to the console: card.querySelector('.rank-badge')?.textContent = rankings[symSlug] ? 'Ranked' : 'Not ranked';

## user

It works but there is a SVG load error (404) for every picture, maybe we have the urls wrong?

## user

Still seeing e.g.

SVG load error for output/zap-U26A1/attempt_0.svg Error: HTTP 404

loadSVG http://localhost:9876/rank.html:374

## user

No more 404s but I still don't see the images and am getting this error

SVG parse error for output/fleur_de_lis-U269C/attempt_2.svg TypeError: can't access property "maxWidth", svgEl.style is undefined

## user

I can barely see the shapes, it looks like they are drawn in a thin dark gray line on a black background, can you make them thicker and add contrast please

## user

OK great it works. Can you clean it up by making it so we see only one set of images at a time, and have a navigation panel on the side that lets us move through the different symbols

## user

Right now the UI is a bit clumsy. I want to be able to sort them as fast as possible, can we modify it so the initial images are displayed and can be dragged into a row of slots below that correspond to their rank order?

## user

Looks good but the drop part of drag and drop is not working, I can drag the cards over but they don't actually stick on the bottom

## user

It still doesn't work. Could I be doing something wrong? Is there a way to add some debugging so we can see what state it is in an confirm it knows when it's being dragged, when it's over a slot, when it's released?

## user

[CARD-4] dragstart, draggable: true rank.html:641:15

[CARD-4] setData/getData: 4 rank.html:645:15

[SLOT-2] dragover 130 rank.html:595:15

[SLOT-2] dragleave rank.html:601:15

[SLOT-1] dragover 387 rank.html:595:15

[SLOT-1] DROP rank.html:605:15

[SLOT-1] data: 4 rank.html:610:15

[SLOT-1] -> already placed rank.html:621:17

[CARD-4] dragend rank.html:649:15

## user

It placed them now I think, including updating the border style, but it just doesn't put the picture where it was placed and keeps it in the top row

[CARD-3] dragstart, draggable: true rank.html:641:15

[CARD-3] setData/getData: 3 rank.html:645:15

[SLOT-2] dragover 12 rank.html:595:15

[SLOT-2] dragleave rank.html:601:15

[SLOT-1] dragover 220 rank.html:595:15

[SLOT-1] DROP rank.html:605:15

[SLOT-1] data: 3 rank.html:610:15

[SLOT-1] -> PLACED attempt 3 rank.html:619:17

[CARD-3] dragend rank.html:649:15

## user

They disappear when plaed but don't actually show up in the bottom squares. And they can't be dragged back to the top. I want it to be possible to move them around or back up after the've been placed, and to see where they are placed

Laguna Poolside 2.1 / Pi

## user

The overall look and feel is good but right now the images are stacked vertically, making it very difficult to sort them as they go off the bottom of the page. Can you make then side-by-side?

## user

I want to be able to sort them as fast as possible, can we modify it so the initial images are displayed and can be dragged into a row of slots below that correspond to their rank order?

## user

One small thing, is there currently a way to export the current state as a json? Can you add one?

Meta Muse Glimmer / Pi

## user

I want to be able to sort them as fast as possible, can we modify it so the initial images are displayed and can be dragged into a row of slots below that correspond to their rank order?

## user

Ok looks good but a bit of cleanup: right now the squares can only be dragged by the bottom border, can we make it so we can drag them by any part? Also, the presentation looks good on the top part, we see the pictures nicely and all five fit across the page. On the bottom row, the sorting slots are bigger and wrap so only three fit on a row, and when the image is dragged into them it becomes smaller and distorted.

## user

Looks like pi froze on the test

## user

There's something wrong with the server, it doesn't respond, I checked it manually as well

## user

That worked. Now can you make it so it saves the state automatically without needing to press `Save Ranking`? Right now if I sort them and click `next` it saves, but if I move manually to another on the nav bar it does not

## user

Great. Now I noticed a few of the svgs don't draw and an error message is shown in the pane. For example crescent moon attempt 5 and umbrella attempt 1. The message says `This XML file does not appear to have any style information associated with it`. I have rendered these elsewhere and they worked fine

## user

Now it just shows the name, e.g. `Attempt 5` in the square, it doesn't render the image

## user

OK that's fixed. Now can you make it so the user can bring the images back to the starting row once they've all been sorted. If I start sorting, I can still bring some back to the original row, but once they've all been moved to the ranking row, the original row disappears and I can only drag them within the ranked row. I'd prefer if the layout stays consistent even when they've all been brought to the bottom row

Claude Opus 5 / Claude Code

## user

OK looks mostly good. I'd like to be able to drag e.g. one to fifth place immediately, or more generally move them between any two spots, either from the unranked to ranked row or within either row, just to make it easier to play with the rankings. I like the click approach, that would normally be the way to speedrun it but it should also be flexible.

Editing

Z.ai GLM 5.2 / Pi

## user

dataset.json contains a list of 100 python code snippets (functions) that have errors introduced. Considering only the code and the "error display" that shows the error python would generate, write an app that a human reviewer could use to confirm the error by adding an annotation of the location and updating the error message if necessary, and edit the code to make it correct. This of this as a test for the human to use pythons error message to confirm what is wrong and make it right.

## user

There are some scroll issues, the main panel appears blank at the top, and when I scroll down I see "loading" and then scrolling much further down I see the actual annotation interface. We should just see the annotation interface in the main panel without having to scroll, once it has loaded

## user

OK, that fixed it. However, the sidebar can only scroll when the whole page scrolls. Please make it scroll separately so the annotation workflow remains on screen in the main panel but the user can scroll the sidebar. I think this was an underlying reason for why previous problem.

## user

OK great. Now let's hide the error category, as well as the error description at the top from the user. The user should only base their annotation off of the python error message and the code

## user

OK, two things I see: What is the user supposed to do with "annotation status: pending?" can't they just modify and save, right now it could be modified but then they need to separately change the status, that feels confuising. Second, what purpose does "run code" serve? It can only run the original code, for which we already have the error message. This could check the message if it was in quesiton, but the messages are not in question, we assume they are accurate

## user

That's fixed. What is the meaning of the tags in the top pane? They seem only to say "IndentantionError:syntax error" or "SyntaxError:syntax error" which isn't super helpful. I guess it's just listing the python error class, but since they are basically all SyntaxError (except IndentationError which python treats separately for some reason) they are not at all informative

Qwen 3.6 35B A3B / Pi

## user

I need port 8765 for something else, can you run it on 8764 please?

## user

Can you add a sidebar for navigation that lets the user scroll through the samples?

## user

OK great. Now let's hide the error category from the user. The user should only base their annotation off of the python error message and the code

## user

Looks good. Can you make it automatically save any code changes when edits are made? Right now if I fix the code then click to another page without confirming, the changes are lost.

## user

One last thing, can you make it possible to highlight a set of columns within a row where the error occurs? Right we can only pick the row for the annotation

## user

Can you make it so highlighting automaticall adds the annotation, like if I highlight cols 5-7 it will populate the fields with those values?

## user

That works. Now can you add a way to verify that there are no longer syntax errors in the corrected code?

## user

Small issue, the button starts yellow and says "check syntax" but once clicked it doesn't go back to this state, for example it says "syntax error" and stays that way even if I go to another sample

Laguna Poolside 2.1 / Pi

## user

Can you please change it so the dataset is preloaded for now

## user

OK - seems to load now. Can you add a navigation pane on the left to scroll through and move to any of the items?

## user

There is an issue with the alignment, the sidebar is on top of the main panel. The side navigation bar should scroll separately and the main annotation panel should stay in a fixed spot occupying most of the screen

## user

Looks like you ran out of space before deciding on any action. Please concisely take some action to fix the alignment issue

## user

There seems to be a bug where the main panel doesn't scroll now, so I can't see all the fields. The rest looks good.

## user

A weird error now, the main panel scrolls independently but it's all in a thin horizontal area towards the top of the screen so only about two lines of text are visible at a time and most of the screen is blank

## user

Now it seems like neither the sidebar nor the main pane scrolls? The content and size look right

## user

You went in a seemingly infinite thinking loop and failed to reach a decision. Please complete the request with minimal thinking

## user

OK great. Now let's hide the error category, as well as the error description at the top from the user. The user should only base their annotation off of the python error message and the code

## user

Better take them out of the sidebar too please. Can you just keep the numbers and add the name of the function as an identifier

## user

Did you restart the server, I still see the old interface even when I refresh

## user

Works, thanks. One last thing, when I click `verify fix` I get `Failed to load Pyodide: V.default.parse is not a function`

## user

This is on a mac, what's the right way to refresh? cmd-shift-r still results in the same error

## user

OK now it gets past that error, says it's loading pyodide, then errors with `Verification failed: can't access property "startsWith", result is undefined`

## user

Works!. Now I notice I can only select the line of the error. Is it possible to let the user highlight the columns as well as the row where the error occurs?

## user

That worked. But edits and annotations don't persist when you go to a different item, either by clicking next or by clicking in the nav panel. If I make and edit and then click to another item, then click back, it's gone, same for when I change the annotation.

## user

The fix works for the code changes but not the annotation

Meta Muse Glimmer / Pi

## user

Great. Can you add a side navigation bar where the user can scroll through the samples easily?

## user

It seems like it's not loading the samples, I see the layout including a sidebar but I don't see any of the content and the interface doesn't work. Also, when I kill the server with ctrl-c it doesn't free up the port which is annoying

## user

OK great. Now let's hide the error category, as well as the error description at the top from the user. The user should only base their annotation off of the python error message and the code

## user

OK - a bit of cleanup. When I press `tab` in the corrected code block, it takes be to the next field (the save review button). I'd expect it to add a tab in the editor window

## user

Great. Now with respect to the annotation. Can you make it so the user can click and highlight the location?

## user

OK that works. can you also update the titles shown in the side navigation bar so it shows the function name along with the number as a way to make it easier to navigate

## user

Can you take off the alert than pops up when saving and just automatically save any updates. This will make is faster for a person to work through them

Claude Opus 5 / Claude Code

## user

OK great. Now let's hide the error category, as well as the error description at the top from the user. The user should only base their annotation off of the python error message and the code

## user

I think the verdict is OK for now. Can you remove the SyntaxError/TabError/IndentationError tags as well. I understand the intent but they are basically all syntax errors, which is the point of the exercise, python just has more resolution re errors related to whitespace for some reason, but it's not super helpful to have about 90/100 just labeled syntax error. I think we can just keep the numbering, maybe add the function name as an identifier to keep them more memorable