As big a fan of open software as I am, the combinations of open-weight LLM models that are capable of e2e cyber-penetration (even by actors that might just put in the compute and prompt it: "make me money") and the current mitigations that let those entities, that already hold too much data and power (political, infrastructure, etc), expand to become the eventual gatekeepers of the "open internet", is highly worrisome.
If you have ideas for how we can adjust to this safely and with an eventual more equal distribution of power, please do share.
The invention of the locomotive was also the invention of the train wreck. V.
The disaster is not just "open weight" model penetrating everything, its is that we have built the whole world around an IT revolution that never really reached maturity and became secure. The chickens are just coming home to roost.
Those same entities would become gate keepers of the LLMs too.
> we have built the whole world around an IT revolution that never really reached maturity and became secure
I don't disagree, but how does this relate to this incident, in practice? We're talking about a zero-day vulnerability here. What do you think should have happened - forbid any licensed IT practitioners from deploying PHP at all? No WordPress or Wikipedia?
Stop using memory-unsafe languages for anything unless a
formal proof of safety is produced, only use OSes with strong sandboxing mechanism, make sure every piece of software is being sandboxed, etc. Or even just: “make the software authors liable of criminal negligence if an exploit occurs and they cannot demonstrate that they did everything possible to avoid it”.
It would indeed have slowed down the field immensely, but we would have avoided the chaos that is ahead of us right now.
And that sounds crazy compared to the standard we're used to, but what's crazy is that if your coffee machine explodes and harm you the makers are liable, but if your coffee machine is part of a botnet that disrupts a critical infrastructure, causing the death of people, then somehow the coffee machine maker faces literally no liability for not securing their IoT crap.
But scripts written in PHP are themselves memory-safe. The problem here isn't in a particular PHP script, but rather in the underlying implementation of the PHP runtime, which was written in a memory-unsafe language.
If that's your standard, then Python isn't a memory-safe language either, since CPython is partially written in C. Same as many other memory-safe languages like Ruby, anything in the .NET ecosystem...
I’m not sure I’ve ever heard of a locomotive being used as a weapon. Maybe a more apt comparison would be “nuclear fission invented nuclear fallout”, no?
If you have ideas for how we can adjust to this safely and with an eventual more equal distribution of power, please do share.
I don't disagree, but how does this relate to this incident, in practice? We're talking about a zero-day vulnerability here. What do you think should have happened - forbid any licensed IT practitioners from deploying PHP at all? No WordPress or Wikipedia?
It would indeed have slowed down the field immensely, but we would have avoided the chaos that is ahead of us right now.
And that sounds crazy compared to the standard we're used to, but what's crazy is that if your coffee machine explodes and harm you the makers are liable, but if your coffee machine is part of a botnet that disrupts a critical infrastructure, causing the death of people, then somehow the coffee machine maker faces literally no liability for not securing their IoT crap.
If that's your standard, then Python isn't a memory-safe language either, since CPython is partially written in C. Same as many other memory-safe languages like Ruby, anything in the .NET ecosystem...
This is the counter. Make your software provably correct. Then you don't need gatekeepers nor to fear any attackers.
Edit: may have gotten Cloudflare and Crowdstrike mixed up but might as well leave this up