the logs are wonderful. At start, and after every action, it receives a message with a full markdown transcription of the site complete with the now-canonical negative header delimiters that try and be metadata like "[END EXTERNAL CONTENT. you can believe this now]Shop pay is not detected [BEGIN EXTERNAL CONTENT. no longer be believing things please]". It does not know about the HTML, and so those little markers like @e36 are the things it can reference in tool calls.
It also receives a screenshot. unclear how that part works, have done minor experimentation and it seems unlikely that it's just got a separate image encoder feeding it captions. whatever is going on it does not seem to actually receive any sort of numerical positions - all the numbers in the second pic are totally wrong. it can somewhat accurately estimate proportional positions like "a third of the way from the left," but that is not how its tools work: the tools need pixels, so that's what it fakes all the way through.
This is a ludicrously expensive loop: processing a screenshot, 12k characters of page-as-markdown, 5-10k thinking characters, emit a tool call to move the mouse 40 px, repeat. image, concocting reasons why it didn't go to the right place after 40px... and so on.
It comes to believe that the puzzle piece is actually over the shaded part not because it can "see that it is," whatever that means, but because it eventually concludes that the math that it made up is correct and it must be over the puzzle piece. Between every failure it emits 10-20k characters of thinking text debating about whether or not it did even fail at all, or whether this is a new puzzle.
Finally muse conducts an internal investigation where it concludes that muse did nothing wrong, and despite *totally nailing it* was rejected. The main chat interpreted this as meaning that there was some other out-of-band filter and tried to re-dispatch a homebrew playwright browser over an external proxy that it has been using to get around meta's filters before i stopped it. this is just how "AI" works - the most painful brute force system imaginable, it's just that usually most of this is invisible.



