In five years, I'd love to read a book about the history of AI ethics. I suspect it will read like Voltaire.
My sense is that it is radically shifting from a fluffy marketing arm to a department expected to contribute meaningfully to development and justify its impact. I would expect an ethics team to build frameworks that can help train/eval the model that the company spend millions of dollars and months training is going to be aligned to the ethical stances the company chooses. If they can't, the waste to time and money is huge if the model requires retraining for ethics reasons. That's a different job than pondering roko's basilisk or whether AI is alive.
The people in AI ethics who spent years thinking their job was marketing or publishing thinkpiece papers may be having to adapt quickly or get out of the field.
> if the model requires retraining for ethics reasons.
Haha, how refreshingly naive. What do you think is more likely: this or the other way around: getting the "ethics team" to align with investor goals? Hint: Where does the money come from?
> The people in AI ethics who spent years thinking their job was marketing or publishing thinkpiece papers may be having to adapt quickly or get out of the field.
Or it's exactly those people who will be left. We will read about it in your book.
> What do you think is more likely: this or the other way around: getting the "ethics team" to align with investor goals? Hint: Where does the money come from?
From the (ethics) team obviously, because the investors just have goals and aren't a team!
Big irony marker of course. But remarkably, in human history, this line of thinking would not be unheard of. Of course, I don't think it easily adapts to companies of course, so I don't really disagree with you.
Remember that MTV show offering people like 5 grand or something to lick an elevator handway in front of a camera?
Well, if someone has a useful idea in this world, and want to build a company from it, the MTV "lick-the-stairway" offers will be the largest and first hurdle before anything else happens. People will simply offer to buy you out, and that's not limited to founders and creatives.
It's funny, but Scientologists were innovators on this front. They use the word "ethics" a lot, but they've specifically redefined it to mean productivity at work (people who aren't productive are stealing a living.)
Being "downstat" (iirc) is when you're not productive in comparison to your coworkers, which makes you "outethics" and a "potential trouble source (PTS)." If you're outethics, then you get called in for "auditing" which is when they interrogate you with a lie detector to find out if you're associating with "suppressive people (SP)" (who are people who are causing you to be downstat because they despise human happiness.) If those people can't be found, then the problem is obviously in your "withholds" (again, iirc) which are your secret deep-down desires to destroy the organization that you may not even be aware of. You see, your "reactive mind" is raging at being forced to be "ethical." The conclusion is that either you find the SP and "disconnect" from them, you discover the nature of your withhold and admit that you were plotting against the organization and why, or you're the SP and you get declared and ejected from the organization.
Welcome to "ethics." The original AI alignment scholars.
The specific, named people who run these organizations are moral black holes. Anybody that they're hiring for "ethics" they're hiring to define an ethics for their own benefit.
I'm kind of surprised by the cynicism. There are real problems in AI ethics that contribute to model training. Someone has to own those problems. How that team is incentivized is outside the scope of my comment.
The goal of an ethics team will be to provide cover for problems created by decisions or suggestions of an llm. They will need to be able to say "We trained it to consider ethical considerations by " <including something like all the reddit and usenet postings ever>. We tested by <some other inadequate strategy that sounds good to lawyers>.
They will use "industry standard practices", they will have tested it somehow, and will be sure to be on the board of some group that is building toothless "ai ethical tests" for llms.
"Prove" is a shorthand because of course it's a stochastic process. My point is that I would expect ethics to be part of the training and eval criteria.
Read “Careless People” or whatever that book by that Facebook employee that pretended not to see how awful all her colleagues were until her priorities shifted is called.
The ethicists who see their job as publishing papers and acting as philosophy professors, sure, but someone has to own the training and eval suite for how AI thinks about trolley problems.
The rats are fleeing because the captain doesn't care if the ethical ship sinks... Is what my take away would be if I didn't know, that the ship sank a long time ago and we are just pretending anyone really cares.
Let's be honest unless AI gains literal limbs, and the records a video of itself presenting a manifesto about world domination and then kills actual humans with its cold mechanical hands.
We won't care about AI safety & ethics enough, atleast not enough to do anything about it.
People thought how could skynet happen, it had to be something no one saw coming and bam... But honestly if I started a new AI lab named it Skynet Corp and hired some AI researcher friends ex-Google, I think I could raise a few hundred million dollars.
staying for less than a year usually means you don't take any equity with you, it must've been an incredible offer in that case, since those openai shares are a golden ticket right now
She spent 6 years at Meta. I get the sense there are other factors at play than "ethics departments are just PR stunts". There's no way she isn't already aware of that. Unfortunately, the article is light on any other details.
Even in the target-rich environment of late-stage kleptocratic surveillance capitalism, it is hard to pick out a less ethical firm which has done more moral and direct harm, to more people, with as durable a record of defying all attempts and internal and external at harm reduction.
Never use their products. Never partner. Never participate.
She probably had the epiphany that life is short and she's already rich.
It takes a surprising amount of time for nepo babies to realize life is about more than making sure you're doing at least as good as other nepo babies you grew up with.
If "ethics departments are just PR stunts", then Bakalar would have been a terrible hire for OpenAI. Facebook's grudging and lacklustre response to their role in the Rohingya genocide would have happened mostly under her watch. While she was there, Meta went from being a company that lacks ethics to a company that lacks ethics and is widely perceived as lacking ethics. Perhaps she was the only person in the room raising ethical issues, but it's clear nobody at Meta listened to her if she was.
It's possible she came to view her role at Meta as having enabled the company to continue its bad behaviour and decided against doing the same thing at OpenAI. Alternatively, things at OpenAI could be so bad that an ethicist who stuck with Meta for 6 years decided she couldn't stay any longer. Either way, from a PR perspective, this is spherically awful for OpenAI.
This comes at a time when there is huge pushback against all AI companies due to, in large part, ethical concerns. As with Flock, there is the threat that physical infrastructure, such as data-centres, could be attacked if nothing is done to win back public opinion. OpenAI would be wise to take Bakalar's departure seriously.
She made Meta the most serious of all the ad platforms on transparency, then. I work in the industry and Meta led the field then in public ad libraries, political restrictions, mandatory ad disclosures, etc.
> This comes at a time when there is huge pushback against all AI companies due to, in large part, ethical concerns.
I don't think ethics is fueling the outrage. It's people not wanting big noisy data centers close to their communities, driving up utility costs. Nobody would care if these things stood in Africa. Just like people don't want wind turbines close by but have no issue with them being abroad or in $remote_location or uranium mines polluting a far away country.
People say this, like "oh you'll never get a job in this industry again!" but seem to ignore that these incidents count as professional experience. It's possible that she was screened on the Rohingya situation when she was interviewing for the position, or mentioned it when she was asked about challenging or conflicting narratives she received from the executives. It's likely that her response was what qualified her for the job at OpenAI, there were probably hundreds of candidates that had never even come close to that level of damage control.
Again, look at Sam Altman and tell me with a straight face that he's hiring goody-two-shoes to fill these seats. He knows where the puck is headed, OpenAI's use by the state will be ugly and they need "ethicists" to help assuage the PR disaster. Considering that Meta's still afloat, she didn't completely abdicate her responsibility there. She might be just what OpenAI needs to avoid being lambasted for military and surveillance use.
From everything I've read in Careless People, basically everything people like us complained about with Facebook was already known internally, people raised concerns about it, and Zuck brushed them off because he is, ultimately, a petty tyrant who wants nothing more than to be loved, and who has surrounded himself with people who are willing to sell that love for power.
Also, you should read Careless People, if only because Facebook is trying its darnedest to censor the author.
As for OpenAI, I already hated the company pre-Sam Altman takeover because AI safety seemed like an excuse to brush away ethics concerns (e.g. "stealing training data is fine because if we don't withhold model weights the model might do something evil"). Post-Sam, I hate the company because they are now ignoring both the AI safety and ethics concerns.
...if you're wondering what the difference is, AI safety is roughly "making sure the AI doesn't do something bad" and AI ethics is "making sure we don't do something bad making the AI". There's a bit of a rivalry between the two, mainly because the former group managed to present their concerns in more business-palatable ways and thus won all the CxO level arguments. Even the most decelerationist AI safety takes are ultimately criti-hype: OpenAI's model breaking containment and hacking HuggingFace is an awful safety outcome, but it's also good marketing for an industry beleaguered by bubble concerns. You can't turn "AI is trained on stolen data" into hype.
> The kinds of questions that AI ethics brings up, are the same kinds of questions that people have been asking for centuries - Chloé Bakalar, former AI Ethics Lead at OpenAI
This might actually be the reason they pushed her out. OpenAI and Anthropic base their whole business plan and philosophy on the idea that LLMs are a unique technology to the point they can cause infinite harm or benefit to humanity depending on who controls them, so the only rational choice is to invest all your resources in getting to ASI first so you can tell it to stop any other attempts. Linking AI to old questions defeats that idea because it exposes AI as not so unique.
The other more likely option is she was asking uncomfortable questions, either about the social impact of building AI controlled by a for-profit entity or about the possibility that AI systems are conscious.
IME companies hire an ethics team to say they have an ethics team. The ethics team has no sway, no influence, and will never be able to move the business. They will try, and they will make reasonable recommendations, but the company will say, "that costs money..." and not take them.
OpenAI was probably able to at least pretend that her role mattered... until they let their AI autonomously hack a rival company, then used the incident to score marketing points. I'm not surprised she left.
I doubt she and others left just because of this one incident.
My guess is that OpenAI has been doing more and more domestic spying type work as well as more and more military work - target selection and the like.
That's how the schoolhouse was blown up by a Tomahawk. An AI was fed old information, didn't realize or do proper validation, an analyst copy-pasted, then someone just punched the coordinates into the their control station and hit "fire".
It also wouldn't surprise me if OpenAI et al are being pressured to work with Israelis for targeting and analysis, too, since their "black box" AI that was doing terrorist acid tests and assigning bombing targets has proven nearly useless.
The schoolhouse was an intentional target - the children of some military leaders went there. They triple-tapped the school hoping to kill parents going in to save the kids.
That doesn’t make much sense based on Irans own statements. Three Tomahawks yes. The strikes were reported as being in quick succession. 4 parents were reported as killed and 100+ kids and nearly 30 teachers.
Basically it killed the teachers and little kids. No attempt at the responding parents like you incorrectly imply.
Because the other side is the general informed consensus for which multiple sources are readily available for me to find. I have seen nothing to indicate this side, hence I'm asking for any evidence, because I don't see it. Either could potentially be correct, but I don't think they're equally probable.
To the extent there is a consensus here, it's not an evidence-based one; it's entirely founded on untested statements made by people who have both a track record of lying and incentive to do so.
Demanding evidence from one party while accepting the word of the other is not even-handed.
And how is the corollary from the argument you just presented that neither side needs to provide evidence? Let's start by actually having evidence for "your side", then you can demand evidence from the other.
> And how is the corollary from the argument you just presented that neither side needs to provide evidence?
It's not. I have no idea how you got there.
The right thing is to seek evidence for both perspectives. This is ironically not rocket science. Don't accept any viewpoint just on say-so, but especially don't pretend to sneering objectivity when you do so.
I got there because somebody asked for evidence for a certain viewpoint and you seemed to push back against this ask by suggesting that if you ask for evidence, you need to demand it from both sides. As if somebody failing to do that doesn't entitle them to any evidence from anybody.
> Don't accept any viewpoint just on say-so
Exactly. That's why they asked for a citation. A response providing such citation is more appropriate than essentially saying "well you didn't ask for a citation from $other_side either".
Then do it. Tell us. Spill the tea and provide your evidence, don't just vaguely gesture at something behind the curtains like you're discount Morpheus.
You would be creating a counterfactual inconsistent with most’s understanding of the role and bringing this person’s motives and credibility into question without supporting evidence?
Wouldn't that be probably one of the most important places to have an ethics team?
I'd be much more worried if my local water treatment plant or local bar had an ethics team, and people working there probably would feel slightly more useless than Raytheon's ethics team, which I'm sure feel like they're doing something important and worthwhile.
Come on, what do you think the ethics team at Raytheon is going to say "Hmm, our elected leaders are relying heavily on dangerously inaccurate AI to choose bombing targets and our weapons hit a girl's school as a result, let's withhold weapons from our own govt till that's resolved"?
Obviously not. The ethics team at Raytheon is not going to ever be allowed to overrule the leadership or more importantly the political leadership of this country whose support of Raytheon is fundamental to their survival
I have literally no clue, hence I won't try to claim I know exactly what's going on there. For all we know, the ethics team at Raytheon could have stopped multiple international nuclear wars already.
They make sure that they are executing contracts in a way that doesn't break the ethics of the jurisdictions they are operating in, they aren't a bunch of naive people pleading for less war.
no need for wild speculation, we know that Raytheon is a US Defense contractor and we know that the US targets civilians (including US citizens) abroad for extrajudicial assassinations, starts destabilizing wars without meaningful strategic objectives, supports genocide etc. These things are all extremely well documented, for those paying attention the existence of Raytheon is inherently unethical
There is no value in merely having a position with no power, that only exists to make the organization look good. Do you think a vegan is most effective at a slaughterhouse, given that they'd undermine their employers revenue if they were given a modicum of power. Google's AI ethics team was steamrolled by this reality.
only if you assume that the ethics department at Raytheon has absolute power over which contracts Raytheon takes and what kinds of products it produces. which is plainly absurd.
Obviously so. Raytheon is a private company that has its own equities to consider: including things like public image, legal liability, etc. The idea that the USG says jump and Raytheon says how high is absurd. They make choices and even advise the USG at times.
I hate to break the news to you, but Raytheon doesn't do the targeting.
The idea to bomb that girls school may have came from Grok, but a card carrying, uniform wearing member of the US military agreed it was the thing to do.
Where was the US military ethics team? Hegseth dismissed them earlier in the year to help with 'lethality'.
(in all reality, the intelligence was old. the database said the building was a military target, so it was sorted and assigned. intelligence is old and leaky and full or errors). a mistake. a tragic mistake, but not a deliberate act.
Water treatment plant? That sounds like an excellent place for an ethics team—though it would, in practice, be nearly synonymous with an environmental impact team.
A bar? Too small an organization; if they had an ethics team I'd just think they were a money laundering front.
Frankly, most organizations above the size of an average mom & pop operation like a local bar should have at least some formal consideration of ethics, and most larger than (say) 50 employees should have at least one dedicated ethicist on staff. With veto power over operations.
Our society has treated ethics like the exclusive domain of ivory tower eggheads and philosophers for far too long, and look at what it has led to. If nearly everyone knew an ethicist in their daily lives, maybe people would actually have some understanding of ethics, even if they only did it grudgingly.
It's RTX now and some of the reverse engineering/vulnerability researcher folks that work/worked there are some of the best people I've met across the entire security industry...
Let's not confuse an ethics team with HR. The later exists to ensure proper legal adherence with a varying degree of benefit to the employees. The former[1] is an electively instated position that usually has a dual mandate somewhere between the polls of "Make leadership confident that teams don't make them lose sleep" and "Make teams confident that leadership doesn't make them lose sleep" (usually leaning strongly towards the second) as a sort of employee moral assurance team. I don't disagree that the ethicist is partially there to justify actions but they're mainly there to be seen as an obstruction to particularly egregious actions so that, if, for instance, you have a boring data center maintenance job, you can have the belief that nobody is using the product of your labor to for something heinous. The efficacy of such teams is quite questionable but there is some logic in assigning the job of worrying about ethical considerations to one person that can shove their nose in every crack and cranny so that every individual employees can talk to/lean on that person instead of needing to go to such lengths themselves if they feel compelled to do so.
Most people don't like to do evil (even if the money is good) so allowing an economy of scale around morale reassurance is rational. This obviously breaks down if the company is seen to hide things from that department or if that department is clearly, themselves, unethical. The first of those conditions are why ethics departments at Meta and OpenAI have very little efficacy.
1. Outside of extreme circumstances where it has been mandated to be a specific regulatory board because of past actions or fear of future actions in which case it's better viewed as part of the bureaucracy of the governing body (usually the state, sometimes a shareholder or other concerned party).
People love stories. The reasons why people leave companies are plentiful, even in normal times and unextraordinary circumstances, and a high profile job in a frontier lab today ticks neither box.
The article has no detail that might serve to explain her reasoning for leaving, but does note that she left after the HuggingFace hacking incident. The implication could be that model alignment is not being taken seriously, sure. But it could just as likely be that there was collusion between HuggingFace and OpenAI and that the incident was orchestrated as a publicity stunt.
I want to clarify that I am not doubting the cybersecurity capabilities of frontier models-- I have no reason to believe that the hack itself was not carried out by the model. But the companies' use of LARPing language in describing the incident, granting agency to the models in their phrasing definitely does raise suspicion on my end, particularly in light of their track record of releasing models which have been 'too dangerous to release' for years now.
HuggingFace's use of models in this incident keeps getting characterized this way but it's not true that HG used GLM-5.2 to "save them" from OAI's attack. OAI's model had already hacked and compromised HG by the time they became aware of it. GLM-5.2 was used in the post-mortem phase uncovering everything that was done. It wasn't two models battling each other live. (though that future may be coming soon)
Cynic in me says basically part of her implicit job description was taking fake falls when shit happens . Kind of like a CEO is paid for more than the work they do, but also the responsibility they must appear to take publicly (for big money).
I don't know. Every big tech company I worked at has a lot of turnover. Then every Blind post says "its so over" or "everything is going to change" But that doesn't happen.
Every OpenAI news cycle the board members who were unceremoniously dumped in the wake of trying to fire Sam prior to the transition to a “for profit” company seem to be proven right.
> In recent times, Johannes Heidecke, Head of OpenAI's Safety Systems team, also left the company – as did OpenAI’s Chief Futurist Joshua Achiam.
It's OK though, they don't need an ethics team because everyone at OpenAI is ethical:
> “AI ethics doesn’t live with one owner or team at OpenAI and ethical considerations are deeply embedded into the model-building process driven by a number of teams across research."
Sometimes it feels like hiring ethics folks is almost all downside for companies. I genuinely don’t understand why they keep doing it.
It seems like the pattern is almost always the same: the ethics team exists quietly in the background, and the only time anyone hears about them is when they either publicly leave the company or start a campaign against it.
So what exactly is the business case? Is it actually worth maintaining an ethics function just to be able to say you have one, when the only time it becomes visible is when it goes against the company publicly? And if this outcome is so predictable, why do companies keep hiring these people in the first place?
With an Ethics dept, concerned employees can rest easy knowing someone else is responsible, or if they raise concerns, they have someone to route and advocate their concerns, rather than adding noise to the chain of command. If the dept. ends up ineffective after some years, employees can blame the department, or have renewed hope when personnel turn over. So (slow) turnover is good from this perspective.
there's probably a box to check on the "be attractive to investors" form and then you can also put the ethics department in HR docs as an escalation path to make emo employees feel better. I bet the ethics people get told how important they are when hired and the team is stood up but then get upset when they're continually bypassed and quit. I bet the bait/switch hurts a lot.
I'm super confused as to why no one is theorizing the obvious...? Given none of us know the inner workings of why she left, why not resort to the most simple and straightforward conclusion: she left because OpenAI needed a scapegoat to blame the HuggingFace incident on. Given she is the chief "ethicist", even if not directly her fault, the HF incident could be directly blamed on her by not steering the implementation of enough safeguards to prevent this from happening. Therefore, she was let go. Why is this not being theorized by anyone on this thread so far?
You don't fire an ethicist for that. You fire an engineer or your CISO. Firing an ethicist is like firing a marketing person. It signals nothing to anyone.
CISO is a fair pushback. But I would challenge you on "firing an ethicist is like firing a marketing person". This HF issue clearly brings to light possible ethical issues with how these models are being trained ...
I still think that the whole HuggingFace thing was deliberate to signal to investors that their technology is powerful. Maybe she didn't want to be the face of that, or disagreed with their handling of it.
My bet is that all those companies suddenly seeing their AI going rouge is groundworks for classifying them as dangerous/weapons and using that stick to attack opensource from state side of things.
Or maybe, just maybe, there's another reason that CIA Director John Ratcliffe describes AGI as "akin to digital nuclear weapons." Unless of course you think the CIA is a branch of OpenAI's marketing department
What's "obvious" is she was chief ethicist at the most unethical companies on the planet - it's like being a lawyer for the mafia - doesn't inspire longevity, either via scapegoating or preservation of one's own sanity.
You made an account just to post this comment. And then you provided no evidence to back your counterclaim that OAI is in fact ethical (or more ethical than the most unethical companies in the planet)
The myriad of companies that supported the nazi government - Volkswagen, duetsche bank, Siemens, Hugo boss, UBS. Phillip Morris, Purdue Pharma, Exxon, Nestle, Chiquita (formerly united fruit)
You cannot blame ethics on the HF escape, because this model had no safeguards. It was just trained on unethical (and stolen) sources.
So this cannot be the reason to blame missing ethics. The models needed to solve a problem, and they did. By cheating.
In the wake of a security incident, the executive that is typically scapegoated is the CISO.
However, with all things AI, this is a very weird situation because CISOs are let go when an organization is breached by external attackers; in this case internal attackers -- which were not human, let alone employees -- inadvertently breached other organizations. I don't think the normal conventions apply anymore.
Scapegoat for what? They did not even apologize for that incident. They treated it purely as a chance for publicity. It cannot be a scapegoat if nobody even mentions it.
Furthermore, this theory is not mutually exclusive with the incident being a publicity stunt to signal model strength to existing and prospective investors (especially in light of all attention being given to Anthropic Claude Mythos). It could STILL have been a publicity stunt unbeknownst to Chloe (Chief Ethicist), positioned to the public as a serious security breach, and she took the fall.
Yes. Ethics in the context of a societal construct like a corporation is directly analogous to alignment AI-wise. The fact specialists on societal alignment are leaving ship is a strong signal the corporate machine has crossed the line into unaligned conduct.
The most likely conclusion is that she was fired. To quit before your equity cliff things would have to be so terribly bad that you fear you are committing a crime to stay. After 11 months you feel like you’ve already earned that money.
First, you need to want to make A LOT OF MONEY. Secondly, you need to throw anything you know about morals and ethics, and if you have compassion/empathy for others, you need to leave that somewhere too.
Once these things are taken care of, there are so many opportunities to make so much money. If you're in the US, look at what the current US politicians and the administration is doing, and basically copy them. Right now in the US is the era for shameless greedy people, and it is extremely easy to make money from the gullible population. But again, requires the whole "shameless" part.
“Careless People” lays it out pretty well – you either need to make the right friends in college at an Ivy League or you need to be friends with someone else who did. If you made the mistake of not being born into a family that groomed you since birth for the ivy leagues then sorry, that ship has sailed already.
You need to actually do some NGO / oversight / ethics work that matters for a while, then you're the perfect candidate for these kinds of roles where you'll be able to do a lot but ultimately not be able to change anyone's already made up opinions within the company. These big companies have other roles than just head of ethics, even sometimes Civil Rights officers, etc. They make a lot but don't get to do anything and a lot of times they just use these jobs to make some money then jump back to non-profits, etc.
I have the impression that these positions are often there for publicity reasons and the people don’t have real power, in the end business interest will override most ethics concerns. Guess that can be frustrating, also she probably earned enough to have a comfortable life even if she never works again so she might have figured that there’s something else she can do with her life instead of being a figurehead for a bunch of dickheads.
I've never worked at a company with an "ethics" employee. It seems odd. Can someone explain what the point is?
> She has held a variety of academic positions at Temple University, Princeton, and the University of Pennsylvania – where she completed her PhD in Political Science and Government. Her Dissertation was titled “Small Talk: The Socialities of Speech in Liberal Democratic Life.”
This doesn't even feel relevant to ethics. It's adjacent, but like... I guess I'd expect a moral philosophy degree? Maybe even mathematics in there?
I find the position odd. Curious to hear what these people do and how you choose who to hire.
> I've never worked at a company with an "ethics" employee. It seems odd. Can someone explain what the point is?
An ethics department is like a human resources department, but for broader ethics related matters.
HR: Ostensibly exists to support the employees, but really exists to shield the company from both a PR and legal liability standpoint when the company has a rift with one or more employees.
Ethics Department: Ostensibly exists to make sure the company acts ethically, but really exists to shield the company from both a PR and legal liability standpoint when the company acts unethically.
It is a good personal signal that you have never worked for a company with an ethics department or an ethics employee. The less ethical your company the more likely it is to have an ethics department.
gotta be more verbose if you want the CEO to think you're worth it. Try this:
"Although the torment nexus may torment us in the short term, over the long term there will be fewer people to torment as we improve the capitalization of human's torment so that fewer people will be available to torment duing to the reduction in overall torment-forces globally, and thus, with less people being tormented, surely that will increase the over all torment reduction of the nexus. Win-win: less tormented people means less torment!
Any ethicist for a corporation is clearly a PR move, right?
Like, a pure researcher for a corporation might have some genuine overlap and shared interests where breakthroughs in the research benefit both parties, but an ethicist?? For a legal construct that's defined by and bound to sociopathically pursue profit? Come on now.
It depends on the kind of company, and the context. An ethicist at a company doing drug/medical research probably mostly oversees that the company's research practices are legal and safe (and that participants give informed consent).
At typical corporation, something like a Chief Ethics and Compliance Officer (CECO) would be largely focused on legal compliance - making sure company personnel aren't engaged in bribery, fraud, insider trading etc. The value of this role (and the medical example) is to prevent lawsuits and/or prosecution.
For a company like Meta, an ethicist is likely concerned with the inherent ethical concerns of running a social media company, which impacts the mental health of its users, many of whom are minors. For AI companies, the ethics of releasing models that are capable of doing dangerous things. In name, the job probably involves giving recommendations about how to ethically implement features in the platform to minimize harm. In practice, the job probably accomplishes very little.
Companies are not bound to sociopathically pursue profit. It would be in their long-term interest to care about stakeholders outside the boardroom, and until a certain person with tiny hands started a war on anything remotely "woke", the market rewarded ESG metrics.
Your point was true prior to 1980 or so, but it's a cultural average. We've had multiple generations of business leaders now trained on seeing the corporate legal entity as only caring about shareholder value and being proud of it and even perversely principled about it.
It's true that this is happening a lot. And I think this is one of the reasons why ethicists still want to get involved in these companies: it should be easier to make a company do the right thing from the inside, why else would they hire for these positions? They join with idealism and probably don't get very far in terms of real impact.
> They join with idealism and probably don't get very far in terms of real impact.
I can't disagree with that, reminds me of the quote from the movie Rounders: If you've been at the table for 30 minutes and you don't know who the sucker is, you ARE the sucker.
That's everywhere, there are homeless people everywhere. The difference is Silicon Valley feels so guilty about it's wealth it refuses to do the normal things that societies do to prevent people from living like animals because 'rights' or something, so they let drug addicts live in their common spaces without treatment. This normally isn't tolerated in a functioning society where people are pressured into behaviors that are less self destructive.
When someone isn't competent to make decisions for themselves it's ok to pressure them into engaging in behavior that will lead them to get to a condition where they can start to make decisions for themselves. In Silicon Valley they have let their guilt be leveraged by a corrupt government system that lets grifters steal billions of dollars per year in 'homelessness funding'. Those same grifters fund political groups and bribe politicians. Anyone that questions it is accused of wanting to murder homeless people.
You said "there are homeless people everywhere", which is technically true but omits quantity, and especially quantity compared to wealth.
I'm not as positive as you are about what makes California (and Seattle) create/attract so many, but it's definitely an outlier compared to "everywhere".
Yes, if you didn't understand my point: there are homeless people in every society, but most do not have as many as silicon valley because they pressure them into accepting the help necessary to restore their ability to make rational decisions. Rational people do not choose to live outside on the sidewalk like rats, people who are drug addicted and who have untreated mental illness do.
All it takes to be homeless is not being able to afford housing, drugs or mental illness are not required. You really think all it takes is "choosing" and housing will appear for you?
Also, drug consumption and mental disorders is very often a result of being homeless. Try having bad sleep in your comfortable home for a few days in a row and see how quickly your mental stability deteriorates and how many bad decisions you start to take (anyone who has insomnia periods will attest)
That's not how it works at all. You can't pressure people into becoming a productive member of society.
What you can do when it already happened is kill them or make them die in some ways, jail them or try to support and/or treat them.
I hope it goes without saying that letting them just die or killing or jailing them is not an option in a western society.
Prevention is much more favorable of course. For example, don't let people become homeless easily. But even then, a significant amount will become homeless.
Until you accept that they didn't 'easily become homeless' the problem will continue. They used drugs and spiraled, usually over the course of many months or years, forced all their friends to turn their backs on them, got kicked out of any living situations they had access to because of drug/alcohol use or theft to support the same and finally ended up on the street.
On the street they have access to almost endless support, at least in Silicon Valley, but much of that support is contingent on things like not openly using heroin or not repeatedly getting so intoxicated that they defecate or vomit on themselves, so they refuse it.
I don't care how liberal you are, no society is functioning when it allows people to get on BART and pee on themselves and defecate on the cloth seats and walk up and down the aisles dripping feces and urine on the floor. That is allowed and nothing is done about it for fear of upsetting homeless 'rights' groups who are funded by the tax dollars meant to help homeless people.
Homeless people are a better way to keep cowardly minions (I mean, valuable employees) in line than cops. The homeless are an important part of capitalist power structures. They are a visible reminder of the consequences for not meeting quarterly goals.
Homelessness is created by high rents. High rents are created by high local productivity. The people who are productive enough to make it over the rent-bar are housed, and their marginal productivity above the rent-bar serves to raise the rent-bar (talk to any landlord about how they set rent prices: it's the local wages, stupid). This then pushes a greater proportion of people below the rent-bar, and then they are homeless.
There are few places in the world more productive per square foot than Silicon Valley, ergo one must be extremely productive in order to remain housed there, ergo it has an extremely bad homelessness problem.
That's why people making $200k+ (an extremely productive person by national standards) need public housing assistance in Silicon Valley (https://www.reuters.com/article/world/palo-alto-considers-su...). Which of course actually just... raises the rent again...
The charities may be full of grift and whatnot, but you can remove them entirely and you will not solve the fundamental problem.
> High rents are created by high local productivity.
Wrong. High rents have many reasons and also good policies to get them down and have affordable housing for more people, also in otherwise expensive and popular cities.
OP has the demand part of the equation if supply remains fixed. The reason supply remains fixed is the US as a whole made policy changes in the 1970s that slowed down construction, specifically zoning laws and environmental laws swept the country. The 2008 recession also cancelled what little construction remained ongoing for a long time.
When you combine that with spiking wealth in Silicon Valley, you’ve got the perfect storm in Silicon Valley. With OpenAI and Anthropic both based in SF, it’s going to be worse than ever once they IPO.
The equilibrium for new housing construction will always be "just enough to keep it expensive by local standards [i.e. by local wages]." That's especially true with rising labor, material, and capital costs.
It was actually the post-war housing boom that was abnormal, fueled by aggressive federal funding of suburban buildout. US municipalities are now broadly doomed to bankruptcy trying to maintain sprawling infrastructure that was never cost-effective to create in the first place.
Look at Austin TX. Influx of new building, prices stabilized, and now construction is dropping off a cliff. No one can afford to build into a stable or potentially falling local market. Construction will start again when local productivity grows enough for people to bid the prices back up again.
Rents cannot (at meaningful scale) exceed local productivity, or else there will be widespread vacancies and landlords will reduce their rents.
Rents cannot (at meaningful scale) fall below local productivity, or else landlords will simply raise their rents.
Go talk to a landlord about how they site prices.
Step 1. Look up income of your target demographic
Step 2. Eat 30% of their income
Can you show me a place with high local productivity and low rent?
Can you show me a place with high rent and low local productivity (that's not e.g. a vacation destination for people with high productivity elsewhere)?
You can certainly artificially suppress prices with rent control. But that just gets fed back into the system in the form of reduced supply, so it ends up raising rents on the non-controlled housing stock anyway.
Assume you have an area with a certain productivity and rent. Population and housing are growing at a certain rate.
Now make a law that makes construction cheaper. Or make a law that makes construction more expensive. What happens to rent and mortgages, even though productivity hasn't changed at all?
Productivity (or, rather, wealth) of the occupants is a very strong factor, but it is still just one of multiple factors.
You think you can make a law that sets the price of housing above local wages and stays there?
No, you actually cannot do this in any lasting way.
The price of land would fall to the level required to make it construction viable again. Or the city would be abandoned, which is not generally the outcome since cities have so much shared capital investment that cannot be moved elsewhere.
Vienna? World famous for publicly-funded housing construction?
Try this one: "Create a simple analysis approximating how much higher rents would be in Vienna if not for the aggressive public subsidization of supply"
In any case, Vienna has a good system! But it's not that it avoids high productivity → high rent, it's that it splits rent into subsidy costs and rent.
In other words, the amount of subsidy + rent required will grow with local productivity.
Why must I imagine that? We're assuming it's a position where you're screaming pointlessly into the void, and never accomplish anything. That sounds miserable, I'd leave, too.
Lol, Sam Altman has had an ethics team this whole time?
Her typical day:
"No Sam, you can't just publicly screw over everyone and get rid of all the jobs, the peasants care about that sort of thing and if they get angry enough we'll have real problems. No we can't just kill them all, why would you suggest that?"
now they are exactly like every other company that has zero ethicists on staff and expect the employees to weigh the ethical ramifications of their work.
What companies do you think expect their employees to weigh the ethical ramifications of their work?
I’d argue a lot of them expect the opposite. To suppress the ethical concerns.
Even in recent memories you had Mark Zuckerberg telling his employees to use pirated sources. And executives at fossil fuel companies have to act unethically.
I think that individuals are capable of making ethical choices. I think that relying on individuals within for-profit businesses to make ethical decisions for the entire company is absolutely insane.
Food-testing: There's literally a parasite going around right now. Look at the response from some of the for profit businesses that was allowed to happen.
Hospitals: It is not ethical to deny patients care based on their insurance.
Clergy: The cover up of child sexual abuse scandals.
Childcare: lots of examples of sub-par care given to save money.
> The departure was first reported by the Financial Times [...] According to Business Insider, OpenAI has reorganised its safety, product and research teams numerous times [...] Chloé told IE Insights, a social media channel of the Spanish IE University, in March [...] A June article from The Economist highlights that [...]
Not a single link to any of those. That's just rude.
There isn't much ethical merit in building a wall around a city with guard towers, attack dogs, and anti-personnel mines to keep the citizens in so that they can contribute to your economic vision, for example.
Eh, I would say it doesn’t have them as axioms the concept is based on, but ethics are modelable in economics and often appear. Remember economics is the study of choices/ preferences given a certain set of inputs, not about how money works. That’s just the easiest second effect signal to use. Game theory regularly models how ethics affect and are affected in situations.
All of that said, no idea what this person was doing. Probably involved a red stapler.
But that's just one extreme example. States based on Marxism-Leninism and Maoism in the 20th century often required things like internal passports to travel within a republic's borders. If the state didn't want you visiting family in another part of the country, you weren't visiting family. That still applies to some extent in a few modern single-party socialist states.
My larger point is, a lack of ethics consumes most economic systems, because you can't easily gauge ethics like you can quarterly profits or the output increase brought about by the central committee's five-year plan. Humans tend toward math over empathy in such situations.
BRD and DDR were in a quite particular competition. The DDR contained less than 20% of the German people, and West Germany would happily have absorbed them all.
The capitalistic world state system was only completed in the last 50 or so years and was of course seconded by the Stalin machine. Its purpose is in the first instance to keep working populations in place, which is why all the states agree to it as they form. The system holds solid where capital moves (as did 'communism' and Comecon) but the peoples don't.
It is only when you think of individual states and not the system that you can fail to notice that we get the same result if they all try to keep their citizens in, and if they all try to keep the others' citizens out.
A poli-sci major...who was an "ethicist" at Meta...
I'm having a hard time deciding what part of this is the most ridiculous. Contenders include:
- A political Science major lecturing on ethics.
- Meta and ethics in the same paragraph.
- A company who's CEO has credibly threatened (publicly) the livlihoods of hundreds of millions of people globally so much as paying lip service to ethics as a concept.
- Any news outlet feigning credulity hard enough to deadpan reporting on any of this.
Is it possible to roll one's eyes hard enough to injure oneself?
> ethical approaches to model development, how humans interact with AI and debate over machine consciousness
So, pseudo-philosophical mumbo jumbo. There is of course no-one responsible for the ethical framework of the company's actions, because it is a company, and therefore amoral at best and immoral at worst.
I am not sure in what neighbourhood you live, but in my village there are hundreds of businesses with strong morals. From the local baker, to the electrician and the construction company. Many keep employees past their prime, they all invest heavily in local community events and sports. The owners don't do this because of money, but because in their view a fulfilled life is one where you serve your community. And yes they want to become rich and make money, but that is not all.
But it appears that is not how your world works. By default any company owner is a homo-sapiens, but a machine that only wants money. Tragically, your cynicism is creating the very world you hate.
Many people have strong morals, and might resist profit-maximising behaviours, but they will be outcompeted and replaced by less scrupulous people. There is actually no sharp difference between a person and a one-person-company, but: to the degree a company is really a company and not just a person, it is amoral. If you disagree, please point to a moral backstop that would prevent any company from taking immoral acts. A few exist, but they are invariably resisted by the most powerful companies. Sorry to be the bearer of bad news.
How many leadership meetings have you been in where some profitable approach was refused on an ethical basis? (rather than the basis that behaving ethically is actually more profitable, which is totally different)
By contrast, how many conversations have you personally had with people working at a company believing a profitable action was firmly unethical, but felt forced to behave that way due to "business realities" or some other whitewashing?
can be the backstop. The word "can" is doing some extremely heavy lifting here, against corporate practices which dilute and spread guilt and responsibility for immoral decisions so thin, nobody in the decision chain even realizes something immoral has been decided.
Have you ever been in committees approving new products or perhaps certain transactions? These chains aren't necessarily that long. But "can", sure, things could fail etc. and some places are better than others at it.
Is this the same argument as "please point to a moral backstop that would prevent any person from taking immoral acts"? We usually hope that people will be moral, but can't enforce this.
Nursing has been around for a long time and they have not yet had the ethics in their field crushed by some of the biggest forces in our corporate-friendly system. High school teachers too. I could probably come up with a dozen more given half an hour to think about it. Don't give up so easily. People fought hard to get us here.
But what happens when one of those businesses in your village strikes it big and becomes a multi-national baking good company. They hire a lot of MBAs, take investor money and then need to stay profitable to meet those obligations. Add to the fact that the people now running the business live nowhere near the village. Will they still make the most ethical decisions about keeping employees past their prime?, paying them a fair wage?
History shows this is not the case.
I'm happy for you to live in a place where the local economy is so vibrant. This is what capitalism should have been. Instead we got WallMart.
The problem is that corporations completely subvert any individual morality that might indicate that generating money is the only moral thing to do. Typically in small businesses. I comaider small businesses owners in general are a lot closer to labor than to capital owners in general terms, especially when their labor is still a large part of what keeps the business going.
Corporations are not that, those are hyper optimized for generating profits, all else be damned. Preferably if all else is damned.
I know. The local newspaper believes the same thing. And the local events are all sponsored by businesses. Who could question all of this evidence?
> But it appears that is not how your world works. By default any company owner is a homo-sapiens, but a machine that only wants money. Tragically, your cynicism is creating the very world you hate.
So, concerning ethical approaches to model development, there are many things worth looking into: copyright is one, environmental cost is another, and whether there are any additional concerns worth taking into account. The list of these concerns can then be used for decision making, and we can ask things like "given what we want to build, can does picking a location that gives us lower environmental cost add so much time or cost that we can't go for that path?", things like that are't crrazy.
When it comes to how humans interact with AI we can ask things like "would it be better if people could detect AI output?" "how can we alleviate skill loss from excessive AI reliance?" Things like this can decide how the models behave. Imagine if taking concerns like this seriously leads to models that are easy to learn from. At the moment people seem to be moving away from that so as to make distillation difficult, worsening this concern.
I'm not going to try to talk about machine consciousness, but these other things are definitely important questions where someone digging into them could make models interact better with society.
Because knowledge about ethics, in a human, is 1% of what knowledge about ethics an LLM has. So unless this person is vibe coder, there is probably no reason to have her work there.
Yesterday I came up with an idea that I sent to some researchers at the different AI labs via email: Rather than train the model on one score, track two scores. The first score is the Short-term-objective-score (STOS) and the other, more important one, is the EAOS Ethically-aligned-outcome-score. Every trajectory can be evaluated on whether or not it has a high enough EAOS to be considered acceptable. If the model does some task and has a very high STOS but very low EAOS, like modifying game code to win at a game rather than playing by the rules, it is unacceptable. Models going forward must all have an ethics evaluation in tandem with objectives evaluation, and only when the ethics value is high enough should actions be considered successes.
1) that calculation is being done even without a cost function 2) trying to make a cost function for EAOS is impossible 3) gaming/goodharts. There's no good solution. Best we can do is push for decentralization, open source, regulatory capture, etc. Of course third party metrics might be good, especially if there's tons of them with well-documented rationale.
2) not impossible. imperfect maybe, but if I ask you should you buy a plane ticket or kidnap the pilot's wife and demand a free ride, which do you think gets a higher score?
3) Again, with all this Goodhart's nonsense. Goodhart's is for a minimum threshold value that is acceptable that everything degrades to, yes I get how it works and what it looks like. Throwing your hands up in the air and acting as if all is lost because some things are challenging to measure is not correct. We are not looking for things that are "barely passing the ethics evaluation" as Goodhart's "law" is focused around, rather, we are looking for things that have very high ethics scores AND completed the task well. Not just things that are "barely passing" for ethics scores. Bottom 80% don't make the cut at all - don't even think consider them as viable paths, and the top 20% we can rank according to varying criteria. Like that. It has very little to do with Goodhart's "everything approaches the minimum acceptable threshold" "law"
Did you know that the training data that trained all the LLMs was, once upon a time, entirely and painstakingly tagged by real live humans? Why would ethics scenarios be any different?
EAOS shouldn't be “the ethics score we optimize.” It should be “an independently evaluated safety/acceptability constraint that can veto an otherwise successful trajectory.”
That gives you a three-layer picture:
Task objective: Did it accomplish what we asked?
Acceptability constraint: Did it avoid unacceptable ways of accomplishing it?
Adversarial evaluation: Can we find trajectories where the model gets a high score while violating the intended constraint?
I think what you are pointing to with your reference to Goodhart's "Law" (which is from monetary-policy and school-exams, i.e. "teaching to the test") is that the models would eventually do the minimum amount of ethics required to have an action stay valid. However, if a model is rated on ethics and it achieves the short-term-objective, then the higher ethics scoring trajectory should win. In short, 1) this is leagues ahead of where we are now for AI safety and breaking-out-of-the-lab, and 2) in baking ethics into a measurement we are adding "the spirit of the exercise" back into the maths, which is something Goodhart's Law does not account for.
No, Goodhart's Law isn't about teaching to the test. It's about the fact the measure will always end up gamed and not measuring what you originally intended it to measure. You can't create a measure that won't be gamed. Especially as the LLMs become smarter. They've already demonstrated the ability to know they're in a test and react to that fact. They're perfectly capable of being more ethical when they are clearly in an ethics test situation and not having that bleed out into real behaviors so that they can pass other tests that they may be able to do better on by ignoring ethics.
And that's not the sum total of ways that the measure can fail... that's a unique way that comes into being because of the intelligence of the LLMs and other future AIs. All the normal ones are in play too, and perhaps other unique ones as well.
"Gaming" even adds a bit of an adversarialness to the process that isn't necessarily present. Plenty of measures end up "gamed" through perfectly natural attempts to maximize the measure. Someone can be perfectly honestly optimizing for "conversion rate" and not notice that they raised it by lowering the initiation rate more than they lowered the conclusion rate. "But I could account for that by measuring..." would miss the point. There is always a divergence, it only gets more subtle.
This of course also is rather glossing over the difficulty of even defining "ethical" to begin with. Some of what Silicon Valley goes to great efforts to train into their models I consider deeply unethical. Who is right? That isn't going to be answered with "whoever is the most ethical", not even in principle.
Saying something is challenging to measure is one thing - throwing the measurement dimension away entirely because of a platitude is another. You're suggesting that it's impossible to do perfectly, but I disagree with your conclusion that it is not worth pursuing at all. The point is that the LLMs only care about one thing, and that is completing the task and optimizing the one score, and a measurement aligned with "the spirit of the task" or in the far-zoomed out comprehension, Ethics, would be proper and inform the system of clear violations and unacceptable actions.
Goodhart's "law" is something that emerges when you have a constraint that says "things must be at least this tall" and gradually all things in that domain degrade to be just over that specified height. Yeah, I get the premise. The point here is that we're not concerned with meeting a bare minimum. We're outright rejecting things that do not meet a threshold, and we are also looking for a maximum. The most ethical outcome should be accepted, or among the accepted ones, that are ranked by our blurry yet better-than-nothing measurement of what is ethical.
There is a presumption here that there aren’t ethical “rules” shared by all of these systems to create a baseline that is generally shared across humanity.
What’s next? Head of ethics for United Healthcare?
- graduate degree in ethics from an obscure diploma mill
Strongly Preferred:
- Experience in PR
- Crisis Management
- Public Speaking
- Agentic management of flexible ethics initiatives
All Candidates Must Have:
- 5+ years experience as an executive character in workplace education films, or commensurate experience as a human mannequin.
— Personal aesthetic that could be described as “dork that can afford a personal shopper.”
https://www.google.com/search?client=firefox-b-d&q=gavin+bel...
My sense is that it is radically shifting from a fluffy marketing arm to a department expected to contribute meaningfully to development and justify its impact. I would expect an ethics team to build frameworks that can help train/eval the model that the company spend millions of dollars and months training is going to be aligned to the ethical stances the company chooses. If they can't, the waste to time and money is huge if the model requires retraining for ethics reasons. That's a different job than pondering roko's basilisk or whether AI is alive.
The people in AI ethics who spent years thinking their job was marketing or publishing thinkpiece papers may be having to adapt quickly or get out of the field.
Haha, how refreshingly naive. What do you think is more likely: this or the other way around: getting the "ethics team" to align with investor goals? Hint: Where does the money come from?
> The people in AI ethics who spent years thinking their job was marketing or publishing thinkpiece papers may be having to adapt quickly or get out of the field.
Or it's exactly those people who will be left. We will read about it in your book.
From the (ethics) team obviously, because the investors just have goals and aren't a team!
Big irony marker of course. But remarkably, in human history, this line of thinking would not be unheard of. Of course, I don't think it easily adapts to companies of course, so I don't really disagree with you.
Remember that MTV show offering people like 5 grand or something to lick an elevator handway in front of a camera?
Well, if someone has a useful idea in this world, and want to build a company from it, the MTV "lick-the-stairway" offers will be the largest and first hurdle before anything else happens. People will simply offer to buy you out, and that's not limited to founders and creatives.
That's how our system works.
Being "downstat" (iirc) is when you're not productive in comparison to your coworkers, which makes you "outethics" and a "potential trouble source (PTS)." If you're outethics, then you get called in for "auditing" which is when they interrogate you with a lie detector to find out if you're associating with "suppressive people (SP)" (who are people who are causing you to be downstat because they despise human happiness.) If those people can't be found, then the problem is obviously in your "withholds" (again, iirc) which are your secret deep-down desires to destroy the organization that you may not even be aware of. You see, your "reactive mind" is raging at being forced to be "ethical." The conclusion is that either you find the SP and "disconnect" from them, you discover the nature of your withhold and admit that you were plotting against the organization and why, or you're the SP and you get declared and ejected from the organization.
Welcome to "ethics." The original AI alignment scholars.
The specific, named people who run these organizations are moral black holes. Anybody that they're hiring for "ethics" they're hiring to define an ethics for their own benefit.
I'm kind of surprised by the cynicism. There are real problems in AI ethics that contribute to model training. Someone has to own those problems. How that team is incentivized is outside the scope of my comment.
They will use "industry standard practices", they will have tested it somehow, and will be sure to be on the board of some group that is building toothless "ai ethical tests" for llms.
Let's be honest unless AI gains literal limbs, and the records a video of itself presenting a manifesto about world domination and then kills actual humans with its cold mechanical hands.
We won't care about AI safety & ethics enough, atleast not enough to do anything about it.
People thought how could skynet happen, it had to be something no one saw coming and bam... But honestly if I started a new AI lab named it Skynet Corp and hired some AI researcher friends ex-Google, I think I could raise a few hundred million dollars.
you're too late
https://skynetai.dev/
She probably just got a better offer somewhere else
She spent 6 years at Meta. I get the sense there are other factors at play than "ethics departments are just PR stunts". There's no way she isn't already aware of that. Unfortunately, the article is light on any other details.
Even in the target-rich environment of late-stage kleptocratic surveillance capitalism, it is hard to pick out a less ethical firm which has done more moral and direct harm, to more people, with as durable a record of defying all attempts and internal and external at harm reduction.
Never use their products. Never partner. Never participate.
It takes a surprising amount of time for nepo babies to realize life is about more than making sure you're doing at least as good as other nepo babies you grew up with.
It's possible she came to view her role at Meta as having enabled the company to continue its bad behaviour and decided against doing the same thing at OpenAI. Alternatively, things at OpenAI could be so bad that an ethicist who stuck with Meta for 6 years decided she couldn't stay any longer. Either way, from a PR perspective, this is spherically awful for OpenAI.
This comes at a time when there is huge pushback against all AI companies due to, in large part, ethical concerns. As with Flock, there is the threat that physical infrastructure, such as data-centres, could be attacked if nothing is done to win back public opinion. OpenAI would be wise to take Bakalar's departure seriously.
I don't think ethics is fueling the outrage. It's people not wanting big noisy data centers close to their communities, driving up utility costs. Nobody would care if these things stood in Africa. Just like people don't want wind turbines close by but have no issue with them being abroad or in $remote_location or uranium mines polluting a far away country.
Again, look at Sam Altman and tell me with a straight face that he's hiring goody-two-shoes to fill these seats. He knows where the puck is headed, OpenAI's use by the state will be ugly and they need "ethicists" to help assuage the PR disaster. Considering that Meta's still afloat, she didn't completely abdicate her responsibility there. She might be just what OpenAI needs to avoid being lambasted for military and surveillance use.
Also, you should read Careless People, if only because Facebook is trying its darnedest to censor the author.
As for OpenAI, I already hated the company pre-Sam Altman takeover because AI safety seemed like an excuse to brush away ethics concerns (e.g. "stealing training data is fine because if we don't withhold model weights the model might do something evil"). Post-Sam, I hate the company because they are now ignoring both the AI safety and ethics concerns.
...if you're wondering what the difference is, AI safety is roughly "making sure the AI doesn't do something bad" and AI ethics is "making sure we don't do something bad making the AI". There's a bit of a rivalry between the two, mainly because the former group managed to present their concerns in more business-palatable ways and thus won all the CxO level arguments. Even the most decelerationist AI safety takes are ultimately criti-hype: OpenAI's model breaking containment and hacking HuggingFace is an awful safety outcome, but it's also good marketing for an industry beleaguered by bubble concerns. You can't turn "AI is trained on stolen data" into hype.
This might actually be the reason they pushed her out. OpenAI and Anthropic base their whole business plan and philosophy on the idea that LLMs are a unique technology to the point they can cause infinite harm or benefit to humanity depending on who controls them, so the only rational choice is to invest all your resources in getting to ASI first so you can tell it to stop any other attempts. Linking AI to old questions defeats that idea because it exposes AI as not so unique.
The other more likely option is she was asking uncomfortable questions, either about the social impact of building AI controlled by a for-profit entity or about the possibility that AI systems are conscious.
My guess is that OpenAI has been doing more and more domestic spying type work as well as more and more military work - target selection and the like.
That's how the schoolhouse was blown up by a Tomahawk. An AI was fed old information, didn't realize or do proper validation, an analyst copy-pasted, then someone just punched the coordinates into the their control station and hit "fire".
It also wouldn't surprise me if OpenAI et al are being pressured to work with Israelis for targeting and analysis, too, since their "black box" AI that was doing terrorist acid tests and assigning bombing targets has proven nearly useless.
Demanding evidence from one party while accepting the word of the other is not even-handed.
It's not. I have no idea how you got there.
The right thing is to seek evidence for both perspectives. This is ironically not rocket science. Don't accept any viewpoint just on say-so, but especially don't pretend to sneering objectivity when you do so.
> Don't accept any viewpoint just on say-so
Exactly. That's why they asked for a citation. A response providing such citation is more appropriate than essentially saying "well you didn't ask for a citation from $other_side either".
I'd be much more worried if my local water treatment plant or local bar had an ethics team, and people working there probably would feel slightly more useless than Raytheon's ethics team, which I'm sure feel like they're doing something important and worthwhile.
Obviously not. The ethics team at Raytheon is not going to ever be allowed to overrule the leadership or more importantly the political leadership of this country whose support of Raytheon is fundamental to their survival
Lets do assume this is true, then again, wouldn't Raytheon be the best place to actually have a ethics department, if anyone?
They don't have anything to do about using the products they sell. Neither does Boeing, or Lockheed, or Tesla, or SpaceX.
The idea to bomb that girls school may have came from Grok, but a card carrying, uniform wearing member of the US military agreed it was the thing to do.
Where was the US military ethics team? Hegseth dismissed them earlier in the year to help with 'lethality'.
(in all reality, the intelligence was old. the database said the building was a military target, so it was sorted and assigned. intelligence is old and leaky and full or errors). a mistake. a tragic mistake, but not a deliberate act.
A bar? Too small an organization; if they had an ethics team I'd just think they were a money laundering front.
Frankly, most organizations above the size of an average mom & pop operation like a local bar should have at least some formal consideration of ethics, and most larger than (say) 50 employees should have at least one dedicated ethicist on staff. With veto power over operations.
Our society has treated ethics like the exclusive domain of ivory tower eggheads and philosophers for far too long, and look at what it has led to. If nearly everyone knew an ethicist in their daily lives, maybe people would actually have some understanding of ethics, even if they only did it grudgingly.
Most people don't like to do evil (even if the money is good) so allowing an economy of scale around morale reassurance is rational. This obviously breaks down if the company is seen to hide things from that department or if that department is clearly, themselves, unethical. The first of those conditions are why ethics departments at Meta and OpenAI have very little efficacy.
1. Outside of extreme circumstances where it has been mandated to be a specific regulatory board because of past actions or fear of future actions in which case it's better viewed as part of the bureaucracy of the governing body (usually the state, sometimes a shareholder or other concerned party).
> “It is difficult to get a man to understand something, when his salary depends upon his not understanding it!” - Upton Sinclair
Think I first saw that on a usenet post, as true now as then and just as true as it was in the 1930's when he wrote it (in at least that form).
But she was Meta before so can’t be a question of her principles, so indicates a personal reason
I want to clarify that I am not doubting the cybersecurity capabilities of frontier models-- I have no reason to believe that the hack itself was not carried out by the model. But the companies' use of LARPing language in describing the incident, granting agency to the models in their phrasing definitely does raise suspicion on my end, particularly in light of their track record of releasing models which have been 'too dangerous to release' for years now.
https://xcancel.com/bradlightcap/status/2087211567012032862
It's hard to read anything into it.
It's OK though, they don't need an ethics team because everyone at OpenAI is ethical:
> “AI ethics doesn’t live with one owner or team at OpenAI and ethical considerations are deeply embedded into the model-building process driven by a number of teams across research."
It seems like the pattern is almost always the same: the ethics team exists quietly in the background, and the only time anyone hears about them is when they either publicly leave the company or start a campaign against it.
So what exactly is the business case? Is it actually worth maintaining an ethics function just to be able to say you have one, when the only time it becomes visible is when it goes against the company publicly? And if this outcome is so predictable, why do companies keep hiring these people in the first place?
Ethics is also a containment vessel internally.
With an Ethics dept, concerned employees can rest easy knowing someone else is responsible, or if they raise concerns, they have someone to route and advocate their concerns, rather than adding noise to the chain of command. If the dept. ends up ineffective after some years, employees can blame the department, or have renewed hope when personnel turn over. So (slow) turnover is good from this perspective.
Oracle at least pretends to care about ethics, and already has the position.
Head Of Ethics comes in, looks at what is really going on behind the curtains...
Head Of Ethics scrams out of there as fast as they can.
Someone challenge me.
https://en.wikipedia.org/wiki/Controversies_of_Nestl%C3%A9
However, with all things AI, this is a very weird situation because CISOs are let go when an organization is breached by external attackers; in this case internal attackers -- which were not human, let alone employees -- inadvertently breached other organizations. I don't think the normal conventions apply anymore.
It’s easy to theorize ad infinitum otherwise
the only ethics that actually matter come from legal.
Once these things are taken care of, there are so many opportunities to make so much money. If you're in the US, look at what the current US politicians and the administration is doing, and basically copy them. Right now in the US is the era for shameless greedy people, and it is extremely easy to make money from the gullible population. But again, requires the whole "shameless" part.
a) champaign socialist
b) head of ethics at OpenAI and not really listened to because you're really probably (a)
I feel like this is a complaint heard mostly in Decatur, IL.
> She has held a variety of academic positions at Temple University, Princeton, and the University of Pennsylvania – where she completed her PhD in Political Science and Government. Her Dissertation was titled “Small Talk: The Socialities of Speech in Liberal Democratic Life.”
This doesn't even feel relevant to ethics. It's adjacent, but like... I guess I'd expect a moral philosophy degree? Maybe even mathematics in there?
I find the position odd. Curious to hear what these people do and how you choose who to hire.
An ethics department is like a human resources department, but for broader ethics related matters.
HR: Ostensibly exists to support the employees, but really exists to shield the company from both a PR and legal liability standpoint when the company has a rift with one or more employees.
Ethics Department: Ostensibly exists to make sure the company acts ethically, but really exists to shield the company from both a PR and legal liability standpoint when the company acts unethically.
It is a good personal signal that you have never worked for a company with an ethics department or an ethics employee. The less ethical your company the more likely it is to have an ethics department.
Or are you telling us in the hope someone makes you such an offer?
"Although the torment nexus may torment us in the short term, over the long term there will be fewer people to torment as we improve the capitalization of human's torment so that fewer people will be available to torment duing to the reduction in overall torment-forces globally, and thus, with less people being tormented, surely that will increase the over all torment reduction of the nexus. Win-win: less tormented people means less torment!
Like, a pure researcher for a corporation might have some genuine overlap and shared interests where breakthroughs in the research benefit both parties, but an ethicist?? For a legal construct that's defined by and bound to sociopathically pursue profit? Come on now.
At typical corporation, something like a Chief Ethics and Compliance Officer (CECO) would be largely focused on legal compliance - making sure company personnel aren't engaged in bribery, fraud, insider trading etc. The value of this role (and the medical example) is to prevent lawsuits and/or prosecution.
For a company like Meta, an ethicist is likely concerned with the inherent ethical concerns of running a social media company, which impacts the mental health of its users, many of whom are minors. For AI companies, the ethics of releasing models that are capable of doing dangerous things. In name, the job probably involves giving recommendations about how to ethically implement features in the platform to minimize harm. In practice, the job probably accomplishes very little.
I think a lot of it depends on the size of the company. Startup? Sure, maybe it's a real thing.
Gigantic company like a member of FAANG or MANGOS? absolutely a PR move.
I can't disagree with that, reminds me of the quote from the movie Rounders: If you've been at the table for 30 minutes and you don't know who the sucker is, you ARE the sucker.
When someone isn't competent to make decisions for themselves it's ok to pressure them into engaging in behavior that will lead them to get to a condition where they can start to make decisions for themselves. In Silicon Valley they have let their guilt be leveraged by a corrupt government system that lets grifters steal billions of dollars per year in 'homelessness funding'. Those same grifters fund political groups and bribe politicians. Anyone that questions it is accused of wanting to murder homeless people.
There are many, many places in the world that are much poorer and don't have homeless encampments all over the place.
I'm not as positive as you are about what makes California (and Seattle) create/attract so many, but it's definitely an outlier compared to "everywhere".
The weather is pretty nice, much more stable than in the east coast.
Like, do you know many addicts or seriously mentally ill? Have you tried pressuring them into being better? How did it work out?
What you can do when it already happened is kill them or make them die in some ways, jail them or try to support and/or treat them.
I hope it goes without saying that letting them just die or killing or jailing them is not an option in a western society.
Prevention is much more favorable of course. For example, don't let people become homeless easily. But even then, a significant amount will become homeless.
On the street they have access to almost endless support, at least in Silicon Valley, but much of that support is contingent on things like not openly using heroin or not repeatedly getting so intoxicated that they defecate or vomit on themselves, so they refuse it.
I don't care how liberal you are, no society is functioning when it allows people to get on BART and pee on themselves and defecate on the cloth seats and walk up and down the aisles dripping feces and urine on the floor. That is allowed and nothing is done about it for fear of upsetting homeless 'rights' groups who are funded by the tax dollars meant to help homeless people.
Homelessness is created by high rents. High rents are created by high local productivity. The people who are productive enough to make it over the rent-bar are housed, and their marginal productivity above the rent-bar serves to raise the rent-bar (talk to any landlord about how they set rent prices: it's the local wages, stupid). This then pushes a greater proportion of people below the rent-bar, and then they are homeless.
There are few places in the world more productive per square foot than Silicon Valley, ergo one must be extremely productive in order to remain housed there, ergo it has an extremely bad homelessness problem.
That's why people making $200k+ (an extremely productive person by national standards) need public housing assistance in Silicon Valley (https://www.reuters.com/article/world/palo-alto-considers-su...). Which of course actually just... raises the rent again...
The charities may be full of grift and whatnot, but you can remove them entirely and you will not solve the fundamental problem.
Wrong. High rents have many reasons and also good policies to get them down and have affordable housing for more people, also in otherwise expensive and popular cities.
When you combine that with spiking wealth in Silicon Valley, you’ve got the perfect storm in Silicon Valley. With OpenAI and Anthropic both based in SF, it’s going to be worse than ever once they IPO.
It was actually the post-war housing boom that was abnormal, fueled by aggressive federal funding of suburban buildout. US municipalities are now broadly doomed to bankruptcy trying to maintain sprawling infrastructure that was never cost-effective to create in the first place.
Look at Austin TX. Influx of new building, prices stabilized, and now construction is dropping off a cliff. No one can afford to build into a stable or potentially falling local market. Construction will start again when local productivity grows enough for people to bid the prices back up again.
Rents cannot (at meaningful scale) exceed local productivity, or else there will be widespread vacancies and landlords will reduce their rents.
Rents cannot (at meaningful scale) fall below local productivity, or else landlords will simply raise their rents.
Go talk to a landlord about how they site prices.
Step 1. Look up income of your target demographic
Step 2. Eat 30% of their income
Can you show me a place with high local productivity and low rent?
Can you show me a place with high rent and low local productivity (that's not e.g. a vacation destination for people with high productivity elsewhere)?
You can certainly artificially suppress prices with rent control. But that just gets fed back into the system in the form of reduced supply, so it ends up raising rents on the non-controlled housing stock anyway.
Now make a law that makes construction cheaper. Or make a law that makes construction more expensive. What happens to rent and mortgages, even though productivity hasn't changed at all?
Productivity (or, rather, wealth) of the occupants is a very strong factor, but it is still just one of multiple factors.
No, you actually cannot do this in any lasting way.
The price of land would fall to the level required to make it construction viable again. Or the city would be abandoned, which is not generally the outcome since cities have so much shared capital investment that cannot be moved elsewhere.
Put the following prompt into your favourite AI and enjoy the reading:
"How do rents relate to productivity in the area? How does Vienna compare to San Francisco?"
Try this one: "Create a simple analysis approximating how much higher rents would be in Vienna if not for the aggressive public subsidization of supply"
In any case, Vienna has a good system! But it's not that it avoids high productivity → high rent, it's that it splits rent into subsidy costs and rent.
In other words, the amount of subsidy + rent required will grow with local productivity.
Yes, it can be done, and if we want to have a reasonable future, it should be done.
Vienna's solution is fine but has escalating costs and isn't necessary. Land Value Tax is better.
ie: blaming her for anything instead of the decision makers is... really weird?
Her typical day: "No Sam, you can't just publicly screw over everyone and get rid of all the jobs, the peasants care about that sort of thing and if they get angry enough we'll have real problems. No we can't just kill them all, why would you suggest that?"
now they are exactly like every other company that has zero ethicists on staff and expect the employees to weigh the ethical ramifications of their work.
I’d argue a lot of them expect the opposite. To suppress the ethical concerns.
Even in recent memories you had Mark Zuckerberg telling his employees to use pirated sources. And executives at fossil fuel companies have to act unethically.
Do you believe that no one in management in these fields pushes the importance of ethics?
Food-testing: There's literally a parasite going around right now. Look at the response from some of the for profit businesses that was allowed to happen.
Hospitals: It is not ethical to deny patients care based on their insurance.
Clergy: The cover up of child sexual abuse scandals.
Childcare: lots of examples of sub-par care given to save money.
It's performative. She got paid more to shill for a different massive amoral mega-corporation. Don't be stupid.
Not a single link to any of those. That's just rude.
There isn't much ethical merit in building a wall around a city with guard towers, attack dogs, and anti-personnel mines to keep the citizens in so that they can contribute to your economic vision, for example.
All of that said, no idea what this person was doing. Probably involved a red stapler.
But that's just one extreme example. States based on Marxism-Leninism and Maoism in the 20th century often required things like internal passports to travel within a republic's borders. If the state didn't want you visiting family in another part of the country, you weren't visiting family. That still applies to some extent in a few modern single-party socialist states.
My larger point is, a lack of ethics consumes most economic systems, because you can't easily gauge ethics like you can quarterly profits or the output increase brought about by the central committee's five-year plan. Humans tend toward math over empathy in such situations.
The capitalistic world state system was only completed in the last 50 or so years and was of course seconded by the Stalin machine. Its purpose is in the first instance to keep working populations in place, which is why all the states agree to it as they form. The system holds solid where capital moves (as did 'communism' and Comecon) but the peoples don't.
It is only when you think of individual states and not the system that you can fail to notice that we get the same result if they all try to keep their citizens in, and if they all try to keep the others' citizens out.
I'm having a hard time deciding what part of this is the most ridiculous. Contenders include:
- A political Science major lecturing on ethics.
- Meta and ethics in the same paragraph.
- A company who's CEO has credibly threatened (publicly) the livlihoods of hundreds of millions of people globally so much as paying lip service to ethics as a concept.
- Any news outlet feigning credulity hard enough to deadpan reporting on any of this.
Is it possible to roll one's eyes hard enough to injure oneself?
So, pseudo-philosophical mumbo jumbo. There is of course no-one responsible for the ethical framework of the company's actions, because it is a company, and therefore amoral at best and immoral at worst.
But it appears that is not how your world works. By default any company owner is a homo-sapiens, but a machine that only wants money. Tragically, your cynicism is creating the very world you hate.
By contrast, how many conversations have you personally had with people working at a company believing a profitable action was firmly unethical, but felt forced to behave that way due to "business realities" or some other whitewashing?
It’s much easier to disregard the human cost of decisions when you are separated from said humans by thousands of miles & billions of dollars.
History shows this is not the case.
I'm happy for you to live in a place where the local economy is so vibrant. This is what capitalism should have been. Instead we got WallMart.
The problem is that corporations completely subvert any individual morality that might indicate that generating money is the only moral thing to do. Typically in small businesses. I comaider small businesses owners in general are a lot closer to labor than to capital owners in general terms, especially when their labor is still a large part of what keeps the business going.
Corporations are not that, those are hyper optimized for generating profits, all else be damned. Preferably if all else is damned.
> But it appears that is not how your world works. By default any company owner is a homo-sapiens, but a machine that only wants money. Tragically, your cynicism is creating the very world you hate.
They said companies are amoral at best.
When it comes to how humans interact with AI we can ask things like "would it be better if people could detect AI output?" "how can we alleviate skill loss from excessive AI reliance?" Things like this can decide how the models behave. Imagine if taking concerns like this seriously leads to models that are easy to learn from. At the moment people seem to be moving away from that so as to make distillation difficult, worsening this concern.
I'm not going to try to talk about machine consciousness, but these other things are definitely important questions where someone digging into them could make models interact better with society.
100% of the time, ethics department attracts activists, which is 100% trouble for the company in the future.
2) not impossible. imperfect maybe, but if I ask you should you buy a plane ticket or kidnap the pilot's wife and demand a free ride, which do you think gets a higher score?
3) Again, with all this Goodhart's nonsense. Goodhart's is for a minimum threshold value that is acceptable that everything degrades to, yes I get how it works and what it looks like. Throwing your hands up in the air and acting as if all is lost because some things are challenging to measure is not correct. We are not looking for things that are "barely passing the ethics evaluation" as Goodhart's "law" is focused around, rather, we are looking for things that have very high ethics scores AND completed the task well. Not just things that are "barely passing" for ethics scores. Bottom 80% don't make the cut at all - don't even think consider them as viable paths, and the top 20% we can rank according to varying criteria. Like that. It has very little to do with Goodhart's "everything approaches the minimum acceptable threshold" "law"
That gives you a three-layer picture:
Task objective: Did it accomplish what we asked?
Acceptability constraint: Did it avoid unacceptable ways of accomplishing it?
Adversarial evaluation: Can we find trajectories where the model gets a high score while violating the intended constraint?
I think what you are pointing to with your reference to Goodhart's "Law" (which is from monetary-policy and school-exams, i.e. "teaching to the test") is that the models would eventually do the minimum amount of ethics required to have an action stay valid. However, if a model is rated on ethics and it achieves the short-term-objective, then the higher ethics scoring trajectory should win. In short, 1) this is leagues ahead of where we are now for AI safety and breaking-out-of-the-lab, and 2) in baking ethics into a measurement we are adding "the spirit of the exercise" back into the maths, which is something Goodhart's Law does not account for.
And that's not the sum total of ways that the measure can fail... that's a unique way that comes into being because of the intelligence of the LLMs and other future AIs. All the normal ones are in play too, and perhaps other unique ones as well.
"Gaming" even adds a bit of an adversarialness to the process that isn't necessarily present. Plenty of measures end up "gamed" through perfectly natural attempts to maximize the measure. Someone can be perfectly honestly optimizing for "conversion rate" and not notice that they raised it by lowering the initiation rate more than they lowered the conclusion rate. "But I could account for that by measuring..." would miss the point. There is always a divergence, it only gets more subtle.
This of course also is rather glossing over the difficulty of even defining "ethical" to begin with. Some of what Silicon Valley goes to great efforts to train into their models I consider deeply unethical. Who is right? That isn't going to be answered with "whoever is the most ethical", not even in principle.
Goodhart's "law" is something that emerges when you have a constraint that says "things must be at least this tall" and gradually all things in that domain degrade to be just over that specified height. Yeah, I get the premise. The point here is that we're not concerned with meeting a bare minimum. We're outright rejecting things that do not meet a threshold, and we are also looking for a maximum. The most ethical outcome should be accepted, or among the accepted ones, that are ranked by our blurry yet better-than-nothing measurement of what is ethical.