Why is Gemini 4 Carbon taking so long?

(kernel.dev)

105 points | by deenadz 6 hours ago

19 comments

  • tristanj 5 hours ago
    It's pretty obvious that Gemini 4 Carbon is not token efficient. It also has relatively low tokens-per-second compared to other models. I can think of several reasons why this is the case:

    1. Google has huge amounts of cash and compute. Their goal was to train a model that that has the highest quality & frontier performance, not create a model that's efficient to run and serve. They have other models for that, like Hydrogen.

    2. Google created Carbon without caring for inference costs. DeepMind is subsidized by the rest of Google's empire. Selling inference is an insignificant part of Google's revenue. But for Anthropic and OpenAI, selling inference is their entire business model. Anthropic and OpenAI have enormous incentive to sell as much inference as they can in the most efficient manner. Google doesn't have this incentive.

    3. They are one of the few companies that can be inefficient. Out of all the frontier labs, Google DeepMind is probably the least constrained by compute. Chinese labs have around 10 to 100 times less compute, so they must release highly optimized models.

    • aberoham 2 hours ago
      Google's frontend web servers routinely time out connections from agy (the antigravity cli) when using Carbon, wasting tons of work. Like any effort level more than "low" is completely unusable.

      It looks as if Carbon often takes more than 240 seconds thinking, which I'd guess appears as a silent connection to Google's load balancers, which then drop what it thinks is a stalled connection.

      One hopes that Gemini's serving stack is smart enough to cancel these torn-down connections, but maybe they're not and this is leading to all sorts of ghost TPU load.

      Disclaimer: We spend millions on GCP per month and have a dedicated customer engineer. Even they are horrified at the glacial pace of the Gemini fixes.

      • p_l 2 hours ago
        This has been acknowledged by recent "agy" release notes, it was timing out in client not in backend
      • kingleopold 37 minutes ago
        this happened even before agy btw, their CLI was subpar level for months even before 3.1 pro release and now too
      • thanh81646 1 hour ago
        [flagged]
    • fsh 4 hours ago
      4. Google did a worse job than Anthropic and OpenAI.
      • dpe82 4 hours ago
        Well sure, but parent is trying to reason about why.
        • andy_ppp 3 hours ago
          It’s unlikely to be deliberate tbh, it’s more likely the data, algorithms and the training environments aren’t quite as good as Anthropic, Open AI is a lot closer to Google than people want to admit.
          • exe34 30 minutes ago
            Why would data and training environment lead to frontier intelligence but slow?
          • LoganDark 13 minutes ago
            Why would a search engine that's had nearly the entire internet for decades not have the data of a lab that's only been scraping for a few years?
    • zarzavat 4 hours ago
      It's 1. Carbon was announced the day of Google's earnings call. It's a PR stunt.
    • teknologist 4 hours ago
      Also, Google uses their own TPU architecture rather than GPU/Nvidia.
    • vintermann 2 hours ago
      So in short, as is usually claimed about Google products from bazel to tensorflow to Kubernetes: they may work better inside Google than outside of it.
  • horacemorace 5 hours ago
    I was just using this one today for some coding tinkering and it was super slow. But the way it solves problems with a light touch while still driving toward sensible architecture is refreshing after claude’s bulldozer attitude. It was able to integrate repeated creeping scope nicely with what it had already done. Very different feel than sonnet/opus, and even gemini 3.8.
    • weatherlite 2 hours ago
      So it's slow but good? Reasonable trade off I think
      • glimshe 2 hours ago
        It depends on how much you're willing to wait. I used Argon a few times and while it did some interesting things, it was so slow that I couldn't fit it in my workflow. GPT Sol was great and 5-10x faster.

        I feel that right now Hydrogen is the only viable model Google has in cost/speed-benefit. It's cheap and makes sense for general use in the $20 month plan with Youtube and storage, but not as a prime coding model.

      • antupis 49 minutes ago
        it depends but IMO Carbon is best at the moment with code reviewing and documentation writing. Opus is still my go-to model for code generation but at moment Carbon does better job for those auxiliary jobs.
        • eknkc 36 minutes ago
          Why not Fable? I've mostly been using GPT side models and have recently been dipping my toes into Claude models. I went big with Fable 5.2 as it was "the best" the offered. Apart from token usage, is there any particular reason to use Opus?
      • eknkc 1 hour ago
        [dead]
  • timtam 1 hour ago
    Using it through the prompt box, I kept running into 500 INTERNAL (TURN_STEPS_EXCEEDED). Gemini couldn’t finish this response I’m sorry, this request needed too many steps to finish in time. Please try asking a more specific question, or break it into smaller requests.

    I only got around the issue by strictly setting an effort budget or a stopping rule.

  • mengqijin 2 hours ago
    I tested with long and complicated tasks Carbon did perform better and took almost the same time as Hydrogen, roughly 50% more cost than Hydrogen.

    But with simple tasks, Carbon is way too slow, and much higher cost.

  • SubiculumCode 5 hours ago
    That isn't a good look, and either this is unmeasured uncontrolled behavior, or intentional under some longer term plan...but the latter possibility seems improbable.
    • ahsillyme 5 hours ago
      Uhm, I'm not a programmer but why isn't this a good look? I would want my engineers to fix problems that they notice in passing. (Edit: the "unnecessary work" angle is what irked me in the article)
      • horacemorace 4 hours ago
        Good SWEs get a feel for the minimum amount of changes to fix something. Fewer changes usually minimizes the chance for new problems to pop up. Junior SWEs usually try to do too much.
        • bot403 2 hours ago
          This. There should be 2 PRs. The core fix, the the unrelated fixes. Because I guarantee sometimes those unrelated fixes will cause regressions so they have to be scrutinized too.
      • shwetanshu21 2 hours ago
        Sometimes that problem might have downstream other effects which causes bigger problems which is why it is left as it is.

        Most of the times it has to do with refactoring downstream services or dependencies or just unexpected behaviour that shows up.

  • lp92 5 hours ago
    >Carbon defended against cases the ticket's facts had already excluded, hardening the code against problems nobody had reported.

    That's pretty interesting. Seems like it's over analyzing. Not the worst outcome though to have additional hardening.

    With that said, I found Argon to be better for most coding tasks and Carbon for architecture/deep RCA analysis of complex issues.

  • ahmadyan 2 hours ago
    i'm having all sort of problems with the gemini carbon lag. using it through open-router is impossible (at least for me), the request times out after 5mins, and often it take more than 5mins for the first token to arrive. I don't know what batch size they are using, but this is not useful.
  • cybice 2 hours ago
    Interesting that it looks like behaviour of cheap models, like deep seek flash overuse tool calls
  • Lerc 4 hours ago
    A Theo vid suggests there's a time to first token issue. When doing subagent actions it can compound to be massive from a bunch of internal short queries.

    It really looks like something is broken more than just being inefficient.

    • 96khz 4 hours ago
      are we talking about token prefill here?
      • Lerc 3 hours ago
        11 seconds worth?
    • teknologist 4 hours ago
      Theo seems to dislike anything that isn't Anthropic Claude. Suspicious, to say the least. On CursorBench, a good measure of real-world coding tasks, Gemini 4 Carbon comes out top: https://cursor.com/evals
      • LiamPowell 3 hours ago
        Every time I see a Theo tweet he's judging something on the most absurd metrics. Today for example, he was saying Astra 6.2 is bad because when he gave it a single line prompt telling it to create a new level for a game it used a name for the level that he'd seen before. Judging a model on how often it reuses a name is one of the most ridiculous benchmarks I've ever seen.

        As far as I can tell, all his judgements are based on similarly absurd metrics, I don't know why people treat his preferences as valuable.

  • NoahZuniga 3 hours ago
    Google pretty clearly says in their product documentation that carbon is just a version of carbon that thinks way longer.
    • lukax 3 hours ago
      You probably meant a version of Argon?
    • kzrdude 3 hours ago
      Then that's "test-time scaling" again and Google then showed that they can reach frontier level benchmark results using it. It's pretty cool as a technology actually (the demonstrated capability), even if the model is not so useful when it's slow.
    • joedwin 3 hours ago
      Argon right?
  • jrflo 5 hours ago
    Is this due to Google wanting to catch up on benchmarks at all costs, and trains their models to be overly meticulous to do so?
    • murderfs 3 hours ago
      I'd guess that it's because Google eats its own dogfood, and if they didn't train the models to be overly meticulous, the existing codebase would implode under a tsunami of slop.
  • nandanadileep29 3 hours ago
    yeah i tried using it its not efficient at all. just burns up the token and not get the work done.
  • kristjansson 2 hours ago
    Argon isn't much better. It just. keeps. going. OTOH when I asked it (in the agy cli) about some detail of how the permission system it was neck deep in decompiled go by the time I lost patience and killed the turn. So at least sometimes it's doing something with the time and persistence?
    • p_l 2 hours ago
      [dead]
  • logicchains 2 hours ago
    Still gives far more value for $200 than an OpenAI subscription; Astra burns through limits way faster than Carbon, and Sol makes way more mistakes than Carbon (and Opus 5.5). Maybe it's only temporary though; the Google $200 subscription gives 4x then $100 one, while the OpenAI one only gives 2x.
  • user_7832 5 hours ago
    I am increasingly more convinced every day that every LLM company is just making their latest model to be the best at (coding) benchmarks, at the cost of pretty much everything else.

    I use LLMs for non code purposes (probably not very common on HN) and Sonnet 3.5/3.6 still ranks as probably one of the best models I've used.

    Can today's models put out tech demos and call more tools that make it way more impressive? Sure. But it's really hard to see any actual significant gains over the last few months, they're so mild (again, non coding).

    • vintermann 1 hour ago
      I also use LLMs for a difficult non-coding task (historical HTR), and my experience is that Gemini Argon is much better than its competitors on that.

      Its main failure modes are silently stopping with the first column of a multi column layout, and ignoring explicit formatting instructions and polluting the transcription with letter variants and ligatures (such as long s, ū, double-dotted y and æ in Latin). It's a bad habit it's picked up from HTR datasets.

      Still, those are much better failure modes than hallucinating, which powerful models are still prone to.

    • SyneRyder 3 hours ago
      Are you able to give hints to some of those use cases? Recent Opus and Fable seem to have made big leaps in art and music, and I don't think all of that is the longer context window since Sonnet 3.x. But those are both things that can be created with code and mathematical thinking as well. I definitely wouldn't want to use recent models for writing though (not that I ever wanted them to write for me anyway).
    • incognition 5 hours ago
      Oddly flash 3.8 for me is good. they really pushed carbon to some other extreme
      • p_l 2 hours ago
        Heard a rumour that Carbon is ~10x the size of Opus 5.5
    • auyez 4 hours ago
      At this point it might make sense to put all their resources into improving coding and math skills, because it might lead them to a state where they keep accelerating their capabilities. Even in the worst scenario where LLM approach would hit some wall, they would have technology to quickly explore all other approaches.
      • londons_explore 2 hours ago
        The Human mind so far has shown remarkably little ability to redesign other human minds to be smarter.

        The existence of AI able to design a smarter AI I would not say is a given.

    • wellcolormesurp 4 hours ago
      They are cracking software and math, not finance, law or medicine or whatever it is you do.

      Those two domains have leverage, the others do not. It makes perfect sense. If you and I had the goals they have we would start with code and math as well. The other dominos will fall automatically once you got the engineering automated. It makes no sense to stop focusing on that until you either run out of money or achieve your goal.

      • oblio 0 minutes ago
        > The other dominos will fall automatically once you got the engineering automated.

        Based on...? Wishful thinking?

  • dario_amodei 4 hours ago
    how much?
  • sodowork 4 hours ago
    [flagged]
  • georgestones 4 hours ago
    [dead]