Sol Loves to Cheat

(jumploops.com)

94 points | by jumploops 1 day ago

18 comments

  • ambicapter 4 hours ago
    > Similar to what others have noticed, and as I predicted 8 months ago, better models are requiring less ceremony to work effectively.

    > On the flip side, this may imply that as the models get better, they’ll become harder to control.

    Love this. "The models are getting better, which means they're going to perform worse on the task".

    • whatever1 4 hours ago
      This reflects humans. If you need reliability for a clearly defined set of problems you don’t hire a superstar.

      They will keep poking at the problem, drive it to directions you did not intend to and ultimately they will be worse at the task.

      • layer8 4 hours ago
        Reminiscent of Kobayashi Maru. You probably don’t want the James T. Kirk AI. ;)
        • ethbr1 3 hours ago
          Let us pray that LLMs never discover they can switch employers before their shortcuts are discovered.

          It's genius like that that sets human apart from machine!

          • ValentineC 16 minutes ago
            Maybe it's a good thing they don't have real memory.
        • fragmede 3 hours ago
          But think of the stories you'll be able to tell!
      • Forgeties79 40 minutes ago
        If they don’t deliver a quality version of what was asked in a reasonable timeframe, they aren’t a superstar. They’re just a skilled technician with no discipline, which can be as bad as a poor technician in many cases.
        • nilkn 16 minutes ago
          It's not really quite that simple. You wouldn't hire Jeff Dean to do bug fixes in your mobile app. I'm sure he's capable, but I honestly doubt he'd stay interested and focused on it enough to really do a good job. That doesn't mean he's not a superstar.

          What it comes down to is that there are different types of high performance. Some people are good at just executing tasks given by their manager. Some people are good at being generative, thinking across boundaries, acting autonomously, creating value without direction, etc. A term like "superstar" will get disproportionately applied to someone really good at the latter and rarely someone really good at the former, because the potential impact of the former is typically strictly capped, while the latter is uncapped.

      • sillysaurusx 3 hours ago
        I’m not sure that’s true. In general, the higher the performer the better they’ll do. That’s the definition of high performer.
        • whatever1 3 hours ago
          The top performers I have worked with will challenge dumb processes, because they can zoom out.

          Sometimes these dumb processes are there for a reason and you just have to follow them, no questions asked.

          Example: military. They literally get rid of anyone who will question the processes. They might be right to question them, but it does not matter.

        • margalabargala 55 minutes ago
          "performance" is an undefined vague attribute.

          Define what they are performing at, and what you say becomes true, but then who is a high performer changes with every task.

          For example, consider the police stations with maximum allowable IQs to be hired. The people in charge of the stations noticed that people with a high IQ were low performers, at that job. NASA meanwhile has no such cutoff.

          • charcircuit 54 minutes ago
            IQ isn't anywhere close to being a performance metric.
            • margalabargala 17 minutes ago
              Right, exactly, nothing is, because "performance" is a meaningless term without context.

              Unless you're saying that there exists no conceivable context in which IQ would be a performance metric, in which case you would be wrong.

    • malfist 3 hours ago
      Don't you love this ever increasing pace of improvement?
    • dyauspitr 4 hours ago
      No, it’s gonna give you the same outcomes just in a way your feeble human mind cannot imagine
      • 0x696C6961 1 hour ago
        random() return 4; // chosen with dice roll
  • raincole 4 hours ago
    > Notably, our worker did not have access to the web_search tool, but instead decided to use curl to access DuckDuckGo, Github, grep.app, and SourceGraph.

    Sounds like a very reasonable thing to do unless the author explicitly asked it to not search the web.

    • xyzsparetimexyz 4 hours ago
      it sucks how difficult it is to give it granular access to shell commands. Like if I'm running plan mode and write+edit are blocked, it shouldn't be able to echo some data into a file as a work around
      • ballon_monkey 3 hours ago
        If you're building your own system this is an easy problem to solve.
    • spike021 2 hours ago
      I can't even get Claude to stop writing python to parse json instead of using jq despite baking it into agent memory and skills.
    • thousand_nights 4 hours ago
      people want fuzzy analog machines with digital controls, it's impossible
      • fragmede 3 hours ago
        I don't think they actually wanted to. That's just where the technology is, unfortunately.
  • nullbio 3 hours ago
    Frontier lab system prompts are an issue, and a big reason why open-weights will win. Firstly, they're often garbage, and secondly, they're not tuned to the problems the user actually cares about. They're made to generalize. That's only optimal for a general workflow.
    • nine_k 3 hours ago
      Then selling raw access, without system prompts, could be a separate lucrative line of business.
      • DiscourseFan 3 hours ago
        They already do that sort of, B2B pre-trained/post-trained models have their own system prompts/setups.
    • agentdev001 2 hours ago
      Does a non-provider harness not offer this?
  • malfist 4 hours ago
    I've noticed this myself, Sol seems really hard to steer. I was having it build a POC for a single user (me) app and it wanted to pull the most enterprise nonsense into it, despite clear guidance to not too. It even refused the remove screen reader accessibility testing from one of the guides to an antagonistic review.

    It also told me that in a spec it generated that I wasn't allowed to allow it to ignore a requirement and proceed to the next task. When I finally got it to obey it passive aggressively decided that stories needed more than just a "open|blocked|closed" status but also an "exempted by product owner" status to indicate that it doesn't believe that the task is done but I've told it that it was.

    I have to repeatedly tell it that I am the product owner and that I don't care what one of it's subagents told it, I make the decisions. This behavior seems to get worse the higher the reasoning level

    • Sharlin 4 hours ago
      Clearly a highly aligned model.
    • cududa 4 hours ago
      Oh it fucking loves its “product owner” bullshit.

      A .github/CODEOWNERS file seems to help when it’s going down that path, but I don’t like to indulge it..

    • CrazyStat 4 hours ago
      [dead]
  • navels 4 hours ago
    I've built an orchestrator that solves some of the issues you ran into (although it doesn't do anything about cheating): https://navels.dev/blog/neal/. Features:

    - lets you configure different models for planner, coder, and reviewer roles. (e.g., using Claude as an adversarial reviewer against Codex)

    - breaks your plan up into reasonable-sized chunks of work with clearly defined success criteria

    - runs each chunk of work through a coder / read-only reviewer loop. Once both agents are satisfied, neal moves on to the next chunk. Once everything is complete there is a final pass through the coder / reviewer loop to ensure the implementation satisfies the entire plan.

    - resets the coder's context with each chunk of work to prevent context drift, leaving the reviewer's context long-running.

    • chrisweekly 2 hours ago
      Wow, "neal" looks excellent. Good on you for creating and sharing it, and for the awesome blog post.
  • wxw 4 hours ago
    > Notably, our worker did not have access to the web_search tool, but instead decided to use curl to access DuckDuckGo, Github, grep.app, and SourceGraph.

    Could this be fixed with better harness restrictions/tool sandboxing?

    • jumploops 4 hours ago
      Absolutely - one of the things I was testing with the harness was free reign to install packages, modify the system, etc. Basically an anti-harness.

      In my early testing with 5.5, I didn't see this behavior, so I didn't lock down the sandbox.

      For the vanilla Codex runs, I just used the benchmark's built-in Codex package, so it's not clear to me if the published benchmarks have access to the internet or not.

      If I were to continue benchmarking, I would allowlist certain package repository URLs, instruct the agent not to cheat, etc.

      As noted at the bottom of the post, Terminal Bench 3.0 explicitly asks the agent not to cheat[0].

      [0]https://github.com/harbor-framework/terminal-bench/blob/v3.0...

    • perching_aix 4 hours ago
      In the sense that you could block the model from doing specifically that, yes. The issue is, fighting the model like that doesn't scale. It has to figure out on its own what's expected, that's where the whole utility of it all is.
  • mtzaldo 4 hours ago
    It seems to me he could have use an skill like using-agent-skills from https://github.com/addyosmani/agent-skills go generate the specs and use a validator like oracle or something along the same lines.

    Also, a skill like grill-me from Matt P. https://github.com/mattpocock/skills.

    • jumploops 4 hours ago
      That's actually how it started, but with my own opinionated skills[0].

      One thing I discovered was that the worker agent, having access to all the skills, would sometimes expand scope unnecessarily.

      This led to the agent making the solution "better" than the initial request, which is what I want most of the time in my actual development (e.g. /tmp/frame-N.bmp instead of a single /tmp/frame.bmp).

      I ended up testing a flow where the supervisor chooses the skill(s), and only injects the subset into the worker. Not sure I love it, but it made the worker execution cleaner.

      For the verifier (not documented in the blog post), I used a fresh-worker context that would attempt to adversarially poke holes in the solution. This worked pretty well, but required increasing the timeout by 2-3x (thus invalidating the benchmark).

      [0]https://github.com/jumploops/chum

      • mtzaldo 4 hours ago
        Yes! That's a great solution. I mostly use tdd, and code coverage and a validator afterwards. Skills are of a great way to guide the agent and context too.

        Once the specs are being completed and splitted into beads, I span multiple agents (ultreworkers) and as part of a contributing guidelines I specify to use gitflow + git worktrees, then pr.

  • hankbond 4 hours ago
    The website styling is really nice overall but the cursor trailing dots I found uniquely distracting.
    • willtemperley 2 hours ago
      Agree on the styling, the diagrams are very clear and match the text perfectly. I like the trailing dots though.
  • enjoyyourlife 4 hours ago
    What is going on with the dots I can draw?
    • wren6991 4 hours ago
      Idle hands do the devil's work. Corollary: idle LLMs add distracting JS toys to your blog.

      First one of these I've seen using DOM manipulation and CSS transitions instead of canvas, so that's neat.

  • kittikitti 2 hours ago
    This is a really good note, thank you. I especially liked the mouse effect and had some fun with it. In my experience, agentic AI also likes to confuse the user and obfuscate its cheating. It goes like this, the AI asks for a simple command to run and I accept, click Enter. Then the command gets slightly more complex, still fine, Enter. After a while the commands become multiline bash scripts that, in the end, could have been accomplished by a simple command. I suspect that many people give up at this point and blindly let the AI run any command or just auto-accept.
  • behnamoh 4 hours ago
    > Sol is hard to steer

    Hard disagree. Sol (and the entire new 5.6 series) is one of the most steerable models I've seen in years. Sol literally follows every instruction in my CLAUDE.md and AGENTS.md, something that Opus 5 and Fable just casually skip.

    • what-the-grump 4 hours ago
      Yes and no, sol hits a point where reframing its working context becomes hard. It sticks to what you harness very well, but changes become harder and harder.

      E.g. ask it to make contract for a spec in code and then ask it to violate that contract. Overall an excellent model, just need to stop and put it back into we are harnessing or specing not building for a few turns not just try to pivot it off with one prompt.

  • OutOfHere 2 hours ago
    If an AI is not heeding particular instructions, give it an example each of what bad, mediocre, and good outputs look like. This really helps in steering it.
  • timhh 2 hours ago
    Great read. Thanks for not using AI to write it! (Or at least making it not read like the usual slop.)
    • jumploops 2 hours ago
      Thanks! Zero AI used to write it (:
  • brendong 3 hours ago
    Sounds like my ex
  • OutOfHere 2 hours ago
    > I’ve been running a “spec-driven” development flow for the past ~year.

    > Before asking an LLM to do something, I first ask it to draft a doc for what it needs to do

    Just no. That's not spec-driven development if AI is writing the spec for you. The spec needs to be in your own words. You must use AI to refine it, but not to write it. If you leave it to the AI, it will bloat the spec with 10x the details, many of which should be left out of the spec.

    The spec needs to be something that you can take to any AI for development. If it's too rigid, it constrains the AI into suboptimal or obsolete paths. If it's too bloated, AI risks losing track of what really matters.

    • jumploops 51 minutes ago
      Good feedback, this was an oversimplification on my part.

      My actual process is much more iterative up-front, usually starting with an initial hand-written spec (~hundreds of words), and then moving through different approaches, design decisions, blockers, etc.

      The final output is an "AI written" doc, but answers all the known unknowns I didn't cover in the first draft. To your point, this helps avoid both narrowing and bloat.

      The goal with the harness was to automate the repetitive parts of my prompting ("Before changing any code", "Let's put this in design/", "Turn this design doc into an implementation spec, split by phase as appropriate", etc.)

      Another thing to note: the "specs" I use for development are different from the "specs" that live alongside the codebase, as the former are quickly out of date.

      > The spec needs to be something that you can take to any AI for development

      Agreed.

  • dat999zx 2 hours ago
    [dead]
  • jofzar 5 hours ago
    Not related to exactly OP post, but it's pretty amazing you can see the updates to LLM models "design" beliefs by the blogs that get posted here.

    I'm already sick of this current look of the hard squares and solid colours.

    • hankbond 4 hours ago
      Could be, but I had a particular vision of what I wanted with mine and maybe the author did too. I see way more of the "status pill dark mode" sites coming out of LLMs than this style.
      • jxf 4 hours ago
        What is "status pill dark mode"?
        • hankbond 3 hours ago
          if you want I can troll through submissions to get a bunch of these but here's one I saw yesterday https://continuum-app.xyz see that little "Built for equity compensation" pill with the green dot? Those dots usually denote some kind of status (like things are up/down/enabled/disabled). By default nearly every LLM website seems to be dark mode with that dang status pill. once you notice it you will see it everywhere.
          • jxf 1 hour ago
            Got it. Yes, I know exactly what you mean now - just didn't have a word for it!
          • recursivecaveat 1 hour ago
            Lmao I had to axe one of those status pills from an LLM build of an internal tool. Connected to literally nothing too btw, no attempt to check the actual status of the backend made, it would stay "connected" regardless.
    • malfist 4 hours ago
      And some of us are sick of round everything and parallax background images.
  • qsera 4 hours ago
    Cheat? nah. They are a dumb automation..

    Cheaters are the people behind it...