We need a new rule that mandates every such "I am leaving <AI company> because of <concern>" post to disclose how much equity they have in the company and how much they have already cashed out. Easy to suddenly find a moral compass when you, your kids and their kids never have to worry about working for money again.
Mr Robinson if you are reading this – if you are truly concerned about AI safety share proof of donation of 100% of your OpenAI earnings and equity towards undoing the damage you have done to society during your time there.
In the absence of that this is simply a career pivot into being an AI "influencer" and raising money for a new scam.
This line of hypocrisy-bashing is unhelpful and will only serve to keep people quiet. Of course people in general need to be wealthy to dare speak out against powerful systems and people, especially in the US where money determines your quality of life so strongly.
Would I respect a martyr who sacrificed their financial security to do this more? Of course. But it's important to applaud people speaking out on important topics
The narrative of AI safety shouldn't be controlled by the same people who caused the problem and profited from it. I don't understand why people are automatically treating "OpenAI" on his resume as a badge of authority. I'm not interested in buying the solution from the same person who sold me the problem. We instead need to amplify independent, unbiased voices.
It's bellow the pay fold but he hasn't been there that long in this case. Skimming his LinkedIn, unless he's got family money, he doesn't seem to be independently wealthy.
You are clearly accusing these people of something. Be clear.
Yes, it is easier to have a moral compass when you don’t have to worry about you and your children starving. But that doesn’t imply that moral compass is wrong or broken.
I'm in full agreement. So many cases of this, and many other others such as falling out with management and peers, new more lucrative offer, and so fourth. Its obvious that these people who come forward are not going to suffer for their new found moral compass.
Or just very dubious timings. Like the other guy from Anthropic that was all over the international news. No followers, no post history but a single post blows up "naturally".
Highly suspect trends that can only make one believe it's marketing.
> if you are truly concerned about AI safety share proof of donation of 100% of your OpenAI earnings and equity towards undoing the damage you have done to society during your time there.
I mean you can be truly concerned and also think donating to AI safety doesnt work, or maybe just be a bit selfish. That doesnt make the concern less real. Its easy to read these articles as the author taking the moral high ground and writing it as some sort of way of proving to themselves theyre a good person, but isnt it just as likely that they think providing an inside perspective can do good by convincing people openAI is a bad actor? I think most of these AI insider accounts largely agree with you that theyre not the most upstanding citizens, does that mean we should write them off?
Because the decision to leave the company is largely based upon their sudden, newfound financial security. They may give another explanation but the only thing that has actually changed is the identification of bagholders ready to cash them out.
The other issue is that the narrative about safety within these companies is largely a function of the extreme financial incentive.
As an example, Anthropic was an "ai safety" company that has now produced an AI that fails to listen to basic instructions. If you were concerned about safety, would you produce an AI that was unable to follow instructions? You ask a question, it begins executing commands and doing things.
Safety is product to sell to politicians, not consumers.
I think its pretty easy to solve these problems: Whenever an AI agent commits a crime, the CEO is held personally accountable, as if they'd committed it themselves.
Do you think a law that nuclear meltdowns would send the CEO to jail would have stopped nuclear accidents from happening?
I don't think this takes seriously enough the possibility that said CEO doesn't think the failure mode is likely and ignores it. Plenty of people are willing to take risks of the flavor "heads you win, tails everyone loses".
Yeah, but this is more like if the Smith & Wesson factory had a cannon mounted on top of it that was mostly used for useful things (blasting roads through mountain passes) and then occasionally they happened to blast another factory.
But OpenAI didn't just make the gun, they're also the ones wielding it. Imagine the Smith & Wesson CEO himself was negligent with his own personal gun.
For me the analogy doesn't totally hold up. Suppose the CEO of Smith & Wesson were to host a firing range on their own property without adequate barriers in place to keep stray bullets from hitting neighboring houses, vehicles, and businesses. Maybe that analogy isn't perfect, but seems closer to what is actually happening.
Financial companies have KYC rules and regulations as they are responsible for reporting illegal activity by account holders. I imagine AI regulations would look similar to that.
Huggingface incident was different in that there was no one else to point the blame to. That's why OpenAI apologised, provided data to independent researchers, worked with huggingface etc.
The case will be lot more complicated if someone uses Kimi to hack into a site. Should the person giving agent the command responsible or the CEO of kimi.
The reality is they have to reduce the capability to ensure security. If someone wants more? Then use the product with your identity and face scan at each session.
That’s true, but people don’t typically say “this Smith & Wesson gun killed someone”; they recognize that the person pulling the trigger is responsible.
With LLMs, at least in the cases of internal/test models doing things they shouldn’t, the people “pulling the trigger” are the board and CEO.
Gun companies don't market their guns as sentient beings capable of independent decision making. Nor do they build systems for shooting things that they host and take money. Gun conpanies are very clear who is in control and where their responsibility ends.
People just want to give up all responsibility these days. If you use the model to do harm to someone else or commit a crime I would think it a lot more reasonable that you be held responsible, instead of making yourself the victim and blaming the manufacturer.
The common timing is bugging me. The trajectory doesn't seem to have been surprising over the last year, so why these exits now? Hey, anyone on the inside, did y'all secretly figure something out, got a computer god locked in the basement? Are rats fleeing a sinking ship? Please share with the class.
Some of these stories are similar in nature to people escaping <insert cult-like religion> once they realize whats actually going on. Alignment to a company's mission is good but it shouldn't be followed like a religion.
Why do people working in tech consistently get disillusioned into some company's mission statement or the equivalent? Its easy to just say the simplest reason is money, but this has been going on for decades though. You don't see the same attraction to adult entertainment (gambling, video, etc.) software jobs so there is obviously a line a lot of people won't cross. Those industries are at least honest about what they do, its not hidden behind some mission statement.
By all indications the shallowest reasoning is once someone can "cash out" thats when their values matter more. Maybe there is an element of maturity that happens after working for 5+ years that kicks in? Maybe it really is achieving FU money? It would be interesting to hear honest accounts from people that went through that cycle across more industries than AI.
I remember something very similar when there was a sudden rush of articles and movies like "The Social Dilemma" criticizing Facebook and social networks, heavily featuring ex-employees, all of them happy to leave with big brands on their resumes and a hefty increase in net worth, all of a sudden having a "worried" expression about what their past employers were doing, as if they didn't know. Same with that book "Careless People".
I sense that these are people who have already eaten the cake and want to somehow absolve themselves of it.
There are numerous problems with “alignment.” What are “human values” to begin with? He outlines some at the beginning of the post, implicitly: build bigger, better, more powerful things faster without adequate safeguards. We are literally pouring trillions of dollars of value into this enterprise, and I would say this is something that many humans also value in a qualitative sense. Then we have explicit values which in the West are largely rooted in Christian morality. Nietzsche circled this dichotomy two hundred years ago and I feel like what we have gotten since then is an increasingly detailed anatomy of power as the basis for what is normal vs deviant behavior. He who has the power, makes the rules, to be reductive.
I do think this carries some weight from this particular author due to the length of his tenure. I happen to agree with him in spirit, but this is still largely a post revolving around sentiment not substance. Does anyone think that the overriding incentives even leave room for something like this in practice?
It is no surprise I guess that the "move fast and break things" culture is itself misaligned with developing potentially highly dangerous technologies. Safety culture and risk aversion are very different of course.
Is this the first time we have been in this position? Can anyone think of some prior examples?
"And second, before we create systems significantly more capable than the ones we have today, we need new science to ensure that more capable models (and their successors) will make safe choices when we aren’t looking." – Which science will never materialise because with blackbox models reaching an opaque optimisation peak one needs to first build the model and track its behaviour before being able to properly understand it and mitigate the risks.
I use opus 5.5 and chatgpt to create PowerPoint, opus really follow the instruction and their PowerPoint generator really well, while chatgpt struggling to even create basic shapes.
I use ChatGPT Work mode all the time to create presentation files. I use Max thinking mode for it. You need to tweak your prompt to get a good result. It took me a week to tweak it, but now it works.
> Two changes are urgently needed. First: AI companies need to rely more on the safety expertise that already exists in other fields. And second, before we create systems significantly more capable than the ones we have today, we need new science to ensure that more capable models (and their successors) will make safe choices when we aren’t looking.
He mentions farther down about learning from aerospace engineers and nuclear engineers about safety. Those industries are heavily regulated, so perhaps regulation above a certain capability level is needed. Defining what that level is might be tough, though.
The second point is harder: in the field of AI, practice has extended far beyond theory, so his call for new science is going to be fundamentally tough, because we can't effectively coordinate a global slowdown in AI development so we can let theory catch up. This means, like so many other industries, the safety lessons will be written in blood.
There are people, who unironically think their way of looking at things is the only proper way and can consider no deviation. And AGI, which knowing how people work, would effectively guide them most of the way, not aligning to their way of thinking is an unacceptable deviation.
That's an interesting point. Maybe a crass comparison, but Dr. Manhattan from the Watchmen comic/movie feels like a worthy analogy to this (obviously fictional though).
What are the concerns of individuals in comparison to the overall progress of humanity?
Always overlooked counterpoint: what point is the progress of humanity if it doesn't take into account the concerns of the individuals?
This pattern is playing out with increasing frequency.
IDK why anyone can’t clearly define “AGI” and why they can’t clearly lay out how we get from our current text-generation algorithms to whatever their idea of “AGI” is.
The cynic in me almost feels like this is staged. An article about culture that is actually an article about how big and smart and scary AI is. I think Michael burry recently said something like “IPOs need hype, calling AI big and scary is hype” in reference to the anthropic IPO.
Last I checked it was still within the laws of physics to run air-gapped systems, and to ensure it is physically impossible for a model to “escape” or gain access to information it shouldn’t have. Maybe this safety guy should have been worried about that and not humble-bragging about writing 12 reports.
This smells more like guerilla advertising. These things are not getting more intelligent, they are still no smarter than a slime mold, we are just burning more power to make slim mold that eats tokens than yesterday
Even though it’s in model producer’s interest that these models do what you don’t want them to do - they want to engineer the model’s to behave in the interests of theirs.
Mr Robinson if you are reading this – if you are truly concerned about AI safety share proof of donation of 100% of your OpenAI earnings and equity towards undoing the damage you have done to society during your time there.
In the absence of that this is simply a career pivot into being an AI "influencer" and raising money for a new scam.
Would I respect a martyr who sacrificed their financial security to do this more? Of course. But it's important to applaud people speaking out on important topics
They are not the same thing, and it’s unhelpful to assume they have no ethics.
>After three and a half years at OpenAI,
Yes, it is easier to have a moral compass when you don’t have to worry about you and your children starving. But that doesn’t imply that moral compass is wrong or broken.
If it wasn’t clear, the coup should have solidified it.
Yet he stayed for 3 more years and vested his stock and improved the company and then spoke out.
I believe that is why most of the comments here are mocking him.
Highly suspect trends that can only make one believe it's marketing.
I mean you can be truly concerned and also think donating to AI safety doesnt work, or maybe just be a bit selfish. That doesnt make the concern less real. Its easy to read these articles as the author taking the moral high ground and writing it as some sort of way of proving to themselves theyre a good person, but isnt it just as likely that they think providing an inside perspective can do good by convincing people openAI is a bad actor? I think most of these AI insider accounts largely agree with you that theyre not the most upstanding citizens, does that mean we should write them off?
The other issue is that the narrative about safety within these companies is largely a function of the extreme financial incentive.
As an example, Anthropic was an "ai safety" company that has now produced an AI that fails to listen to basic instructions. If you were concerned about safety, would you produce an AI that was unable to follow instructions? You ask a question, it begins executing commands and doing things.
Safety is product to sell to politicians, not consumers.
Not serious.
> Easy to suddenly find a moral compass when you, your kids and their kids never have to worry about working for money again.
If someone is in this situation, you can safely ignore their hand-wringing about “safety.”
I don't think this takes seriously enough the possibility that said CEO doesn't think the failure mode is likely and ignores it. Plenty of people are willing to take risks of the flavor "heads you win, tails everyone loses".
This is not the case with SaaS services.
The case will be lot more complicated if someone uses Kimi to hack into a site. Should the person giving agent the command responsible or the CEO of kimi.
The reality is they have to reduce the capability to ensure security. If someone wants more? Then use the product with your identity and face scan at each session.
Trade offs mate.
With LLMs, at least in the cases of internal/test models doing things they shouldn’t, the people “pulling the trigger” are the board and CEO.
Reading posts on here is slowly becoming akin to brain rot.
Why do people working in tech consistently get disillusioned into some company's mission statement or the equivalent? Its easy to just say the simplest reason is money, but this has been going on for decades though. You don't see the same attraction to adult entertainment (gambling, video, etc.) software jobs so there is obviously a line a lot of people won't cross. Those industries are at least honest about what they do, its not hidden behind some mission statement.
By all indications the shallowest reasoning is once someone can "cash out" thats when their values matter more. Maybe there is an element of maturity that happens after working for 5+ years that kicks in? Maybe it really is achieving FU money? It would be interesting to hear honest accounts from people that went through that cycle across more industries than AI.
I sense that these are people who have already eaten the cake and want to somehow absolve themselves of it.
I do think this carries some weight from this particular author due to the length of his tenure. I happen to agree with him in spirit, but this is still largely a post revolving around sentiment not substance. Does anyone think that the overriding incentives even leave room for something like this in practice?
Is this the first time we have been in this position? Can anyone think of some prior examples?
> Two changes are urgently needed. First: AI companies need to rely more on the safety expertise that already exists in other fields. And second, before we create systems significantly more capable than the ones we have today, we need new science to ensure that more capable models (and their successors) will make safe choices when we aren’t looking.
He mentions farther down about learning from aerospace engineers and nuclear engineers about safety. Those industries are heavily regulated, so perhaps regulation above a certain capability level is needed. Defining what that level is might be tough, though.
The second point is harder: in the field of AI, practice has extended far beyond theory, so his call for new science is going to be fundamentally tough, because we can't effectively coordinate a global slowdown in AI development so we can let theory catch up. This means, like so many other industries, the safety lessons will be written in blood.
Yeah so that's never going to happen
What are the concerns of individuals in comparison to the overall progress of humanity?
Always overlooked counterpoint: what point is the progress of humanity if it doesn't take into account the concerns of the individuals?
This pattern is playing out with increasing frequency.
Last I checked it was still within the laws of physics to run air-gapped systems, and to ensure it is physically impossible for a model to “escape” or gain access to information it shouldn’t have. Maybe this safety guy should have been worried about that and not humble-bragging about writing 12 reports.
Yay humanity's future...
Even though it’s in model producer’s interest that these models do what you don’t want them to do - they want to engineer the model’s to behave in the interests of theirs.
I can’t believe people can’t see it lmao.