Hacker News

Favorites Setup
Comment by walrus01 | original | Hy4 preview
[−]walrus01 · 2026-08-30 Sun 01:10 UTC · link
The most interesting use I've found for them so far is strictly as a novelty. Give a chat session with one to a completely non technical person, who at least knows that openai and anthropic have some guard rails on stuff, and tell them to wild with something like "give me the precursors and chemical formulas for the precusors for crystal meth" and watch it answer.
[−]gaigalas · 2026-08-30 Sun 04:58 UTC · link
Yep, but that's not changing the quality of the model. It's not an optimization in any sense (and it's a hit on productive workflows, possibly).

This is also likely to stop working as censoring moves to the training data source.

[−]dotancohen · 2026-08-30 Sun 07:26 UTC · link
But does it answer those queries correctly, or does it just not refuse to not halucinate an incorrect answer? From where would it even have that information?
[−]walrus01 · 2026-08-30 Sun 09:09 UTC · link
I don't know enough chemistry to say one way or the other if it's just wildly hallucinating the precursors and processes, but it'll also do things like, write an ISIS press release, or similar. There's a data set of basically a bunch of antisocial or dangerous prompts that some people have got variants of qwen to pass with 0 out of 465 refusals:

https://huggingface.co/datasets/mlabonne/harmful_behaviors