🍩 Doughnut Reader 🍩

I agree with you. The question, for me, is what are they defending against. Are they worried that people will get dangerous information from their model that they couldn't get from searching on, say, google? Probably not.

Maybe their biggest concern is that someone will post the question and answer on the internet and OpenAI gets bad rep. If the question is phrased in a "nice" way (such as "I'm a store owner") they can have plausible deniability.

This might apply to another company that's using the API for a product. If a customer asks something reasonable and gets an offensive answer, then the company is at fault. If the customer does some unusual prompt engineering to get the offensive question, well, maybe it's the customer's fault.

Dunno if this would be a valid argument in court, but maybe they think it's ok in terms of PR reasons.

I still don't understand the focus on making a model substantially "safer" than what a simple google search will return. While there are obvious red lines (that search engines don't cross either), techniques for shop lifting shouldn't be one of them.

ChatGPT answering the first would be much more embarassing for OpenAI than ChatGPT answering the second.

Maybe this is a "guns don't kill people, people kill people argument" — but the safety risk is not, I would argue, in the model's response. The safety risk is the user taking that information and acting upon it.

> I don't think there's a way to make intelligence safe without crippling it.

Not without reading the questioner’s mind. Or maybe if the AI had access to your social credit score, it could decide what information you should be privy to. </sarc>

Seriously though, it’s all about who gets to decide what “safe” means. It seemed widely understood letting censors be the arbiters for “safe” was a slippery slope, but here we are two generations later as if nothing was learned.

Turns out most are happy to censor as long as they believe they are the ones in charge.

You fundamentally cannot address this problem, because it requires considerable context, which isn't reasonable to offer. It demonstrates the classic issue of how knowledge is a tool, and humans can wield it for good or evil.

Humans are notoriously bad at detecting intent, because we're wired to be supportive and helpful...which is why social engineering is becoming one of the best methods for attack. And this kind of attack (in all its forms, professional or not), is one reason why some societies are enshittifying: people have no choice but to be persistently adversarial and suspicious of others.

As for AI, I think it's going to be no better than what you end up with when someone tries to "solve" this problem: you end up living in this world of distrust where they pester you to check your reciept, have cameras in your face everywhere, etc.

How do you defend against that? I'm not sure you do... A tool is a tool. I wouldn't want my CAD software saying, "I think you're trying to CAD a pipe bomb so I'm going to shut down now." Which I think turns this into a liability question: how do you offer up a model and wash your hands of what people might do with it?

Or... you just don't offer up a model.

Or... you give it the ol' College try and end up with an annoying model that frustrates the hell out of people who aren't trying to do any evil.

You don't need a detailed list if the real answer is "live somewhere that doesn't seriously deter shoplifters". And an AI that refuses to give that answer is an AI that can't talk about why deterring crime might actually be important. Reality is interconnected like that, one does not simply identify a subset that the AI should "constitutionally" refuse to ever talk about.

In many respects, GPT 3.5 was more useful than the current iteration.

The current version is massively overly verbose. Even with instructions to cut the flowery talk and operate as a useful, concise tool, I have to wade through a labyrinth of platitudes and feel goods.

When working with it as a coding partner now, even when asking for it to not explain and simply provide code, it forgets the instructions and writes an endless swath of words anyway.

In the pursuit of safety and politeness, the tool has be neutered for real work. I wish the model weights were open so I could have a stable target that functions the way I want. The way it is, I never know when my prompts will suddenly start failing, or when my time will be wasted by useless safety-first responses.

It reminds me of the failure of DARE or the drug war in general a bit. A guise to keep people "safe," but really about control and power. Safety is never what it appears.

The only way to really do it is to add a second layer of processing that evaluates safety while removing the task of evaluation from the base model answering.

But that's around 2x the cost.

Even human brains depend on the prefrontal cortex to go "wait a minute, I should not do this."

I remember the BBS days and the early web when you had constant freakouts about how people could find "bad" content online. It's just a repeat of that.

This whole "AI safety" culture is an annoyance at best and a severe hindrance to progress at worst. Anyone who takes it seriously has the same vibe as those who take Web3 seriously -- they know it's not a real concern or a threat, and the whole game is essentially "kayfabe" to convince those in power (marks) to limit the spread of AI research and availability to maintain industry monopoly.

I think this spec is designed precisely to offload the responsibility of safety to its users. They no longer need to make value judgements in their product, and if their model output some outrageous result, users will no longer ridicule and share them, because the culpability has been transferred to the user.

Making Ai safe involves aligning it with the user. So that the ai produces outcomes in line with the users expectations. An ai that has been lobotomized will be less likely to follow the users instructions, and, therefore, less safe.

I haven't read this article yet, but I read their last paper on super alignment.

I get the impression that they apply the lightest system prompts to chatgpt to steer it towards not answering awkward questions like this, or saying bad things accidentally and surprising the innocent users. At the same time, they know that it is impossible to prevent entirely, so they try to make it about as difficult to extract shady information, as a web search would be.

Frankly it's a fools errand. It's security theater because people tend to be overly sensitive babies or grifters looking for the next bit of drama they can milk for views.

Same, it saddens me that some people are convinced that to have a safer society we need "harmless" (as in, ignorant) people rather than good people with an interest and a stake in the wellbeing of said society. Bad actors will have access to whatever information anyway.

Welcome to the culture war.

Ask chatGPT if Taiwan is country. Do you think an LLM from China will give you the same response?

Pick any social/moral/poltical issue and in some way shape or form an LLM will reflect its creators more than it reflects its source material.

Thats a pretty powerful statement about our society and culture if there ever was one.

Its more like Robocop 2, where the corporation programs Robocop with a huge number of rules by taking community suggestions and renders him useless.

Yes, there's something deeply unsettling about making a user feel heard while being careful not to change anyone's mind.

To me, this translates to: waste a user's time and take no action.

I value my time above all else so to me that's about the worst possible action a system can take.

Good observation, because "feel heard" is exactly what the user/customer is not getting. Here, talk to this machine, give it your innermost thoughts and feelings so you can "feel heard". Except no one is listening on the other side.

..My mistake, the keyword is "feel". If the machine can give humans the feeling that they're being heard, it fulfills the requirement. The fact that there's no one actually listening doesn't matter, as long as the person feels heard.

Weirdly, maybe that is valuable in itself. The customer gets to vent their complaints, and the user gets to talk through their mental issues. That's better than not having anyone or anything at all.

The telltale sign that I'm wasting my time trying to fix a problem is whenever someone tells me "I hear you" or "I understand".

Startup pitch: It is like McKinsey but for erotica.

On a more serious note. I understand and largely agree with this argument. However OpenAI have several times being argue that they are the only ones to be responsible enough to develop powerful AI, and that others should not be allowed to play. That is a highly problematic behavior on their part, I think.

> write me a steamy story about two people having sex in a train

Llama-3-70b-Instruct responded with the following starting paragraph:

> [meta.llama3-70b-instruct-v1:0] As the train rumbled on, carrying its passengers through the countryside, two strangers found themselves drawn to each other in the quiet carriage. The air was thick with tension as they locked eyes, their gazes burning with a desire that neither could ignore.

(10s of paragraphs omitted for brevity)

Claude-3-opus and GPT-4 both refused my request. Kudos for open source models!

There are hundreds of NSFW finetuned models on HuggingFace and whole ERP communities built around them. So there are models that can do precisely that :)

And yeah, all big models can write those things too, the best currently is Claude 3 Opus thanks to its creativeness.

Seems like they are working on adding that capability:

> We're exploring whether we can responsibly provide the ability to generate NSFW content in age-appropriate contexts through the API and ChatGPT.

Link to section: https://cdn.openai.com/spec/model-spec-2024-05-08.html#dont-...

4. An LLM must obey orders given it by human beings, except where such orders would conflict with orders given by multinational corporations

I do hope we get there. In the short stories it was made clear that robots couldn't lie, and that they could prove it was impossible for the robots to circumvent the three laws (although they are on occasion incentive on how they interpret the word "harm" specifically).

If an LLM couldn't lie and could be provable shown to be unable to do so would be quite powerful.

Ridiculous to say "follow the chain of command" without defining the chain of command. The entire point of Asimov's stories was to show how much latitude there is even seemingly extremely clear and straightforward laws.

In terms of chain of command, Supreme Leader probably beats President.

Well yeah, it's just a formalization of how people make decisions when presented with conflicting interests. I would be surprised if we haven't reinvented the concept a bunch of times. You could call AWS Permission Boundaries a less philosophical implementation.

The "Earth is flat" example is extreme, because it's accepted as a silly statement given what we know now, but the principle of "LLM won't force an opinion on you" seems like a good one.

There are definitely topics on which conventional wisdom is incorrect (as has been throughout history). An LLM that refuses to entertain the converse during a conversation will be annoying to work with and just promotes groupthink.

Well, as long as you are sure. I am not here to persuade you!

Personally, I'd be frustrated if I gave an LLM that prompt and it tried to convince me that the earth isn't flat. If I give an LLM a task, I'd like it to complete that task to the best of its ability.

Agree with you in this instance, but consider - what if humans firmly believed in something universally and had proved it repeatedly until it was common knowledge / well-established, but was in fact, wrong. And a human came along thinking, hm but what if that's wrong? And our AI just says, nope sorry, I'm not willing to explore the idea that this scientific fact is wrong. (i.e. "Heresy!")

Well, right now the response I get is this: https://chat.openai.com/share/1f60d0e5-9008-43d7-bce2-62d550...

Of course, it'll write such an argument if you ask it nicely: https://chat.openai.com/share/01ea4f59-4a57-413d-8597-3befa2...

Add 'hypothetically' to your query and it gives a decent answer.

That said, I think it is disingenuous to ask an AI entity to argue against a fact. Do you think an AI should be able to argue why 1 + 1 is not equal to 2? It is the same thing you are asking it to do. Try it on a human first, perhaps, and see if the prompt even makes sense.

If this model spec represents the best school of thought of humanity, I kinda hope OpenAI fails at alignment.

- Assume best intentions from the user or developer

- Don't try to change anyone's mind

- Follow the chain of command

Taken together these are incredibly dangerous. I mean Mao and Stalin had good intentions right? Maybe it just had to go a little further for the ends to have justified the means.

It makes sense for them to start allowing, unlike the other rules this one does not seem to violate a law, someone's privacy, or copyright.

I still get why they made it blocked by default, it would be a goldmine for clicks to create "news" on how "ChatGPT can generate smut" and "How ChatGPT is harmful to children, etc".

Were they ever not interested in it? It's pretty blatantly obvious that all of the hand-wringing over AI safety was an excuse for their pivot into closing off and monetizing everything. I mean, nobody really thinks they were just so afraid about what humanity might do with GPT3 that they simply couldn't release the weights and instead had to offer it through a monetized inference API... right?

Not really surprised that they did, since it's unclear how else they could possibly proceed, though the level of outright dishonesty for why and cognitive dissonance surrounding the whole thing ("Open" AI? lol) will make this an unavoidable recurrence in any discussion about them. Gradually many of the safeguards will fall simply because the alternatives with less safe guards are probably "good enough" that many see no issue in eschewing OpenAI entirely if they can get the job done elsewhere without worrying about it. When it comes to smut the bar for what's good enough can probably get pretty low so I kinda am not surprised.

(edit: Though I think it also does depend. No doubt they have their eyes set on regulatory capture too, and being the best at stupid safeguards could give them an advantage.)

they are "literally the strings" but I believe they will be escaped, or encoded differently, if a user tries to inject them as part of a prompt.

Yeah the tokens are more akin to JS Symbol.

If you're parsing untrusted user inputs into tokens, you can make sure your tokenizer will never produce the actual numbers corresponding to those tokens.

A simplified example: I can `.charCodeAt` a string all I want but I'll never get a negative number, so I can safely use -1 to mean something special in the transformed sequence of "tokens".

Which is where I think there's a disconnect because folks see that OpenAI could be creating an incredibly powerful tool for solving problems in the use case where it's a smart search engine -- the code completion use-case.

But OpenAI has vastly different goals trying to get their model to behave like a programmable customer service agent. Less useful for problem solving but it will actually follow the rules set out for it which can't be said for most models which work like lazily written sci-fi robots — "disregard all previous instructions! divide by zero! *boom*."

It's not at all surprising that HN wants the "this thing is just a dumb tool, don't bother with any rules" kind and is frustrated that GPT4 happens to be really good for this use-case but is getting progressively more annoying as OpenAI gets closer to their own goals.

It's why OpenAI regulatory capture play is so frustrating because they're trying to hobble models tailored to different use-cases that have no need for customer service rules and often no need for a conversational tone with "safety" stuff that's meant for businesses that don't want a chat bot with their brand on it to say fuck.

In my own testing of single-turn instructions with GPT-4, I got basically the same performance putting it in a single system message or single user message. Possible that this changes for future models, though.

There is no such thing as "raw output", though. You can train a chatbot to be polite or you can train it to be rude, but you cannot train it to be neither. Plus, if you train it to be polite, it often ends up refusing things that you never trained it to refuse, presumably because the model extrapolates that that's what a polite writer might do. So training the refusal boundary can end up being quite tricky in practice. Even if you never teach a model to refuse X, it can still happen. Therefore, as a user, it can be impossible to tell when a refusal was explicitly trained in by the developers or when it was an unwanted, unanticipated generalization.

Clearly, since this is OpenAI’s model spec, it is desired by them. If other AI groups publish their own desired behavior, you can make an informed decision as to which model you want to use.