That makes sense to me! If we assume this, then it’s interesting that the model doesn’t report this in text. Implies something about the text not reflecting its true beliefs. 

Thanks! This is really good stuff, it's super cool that the 'vibes' of comics or notes transfer over to the text generation setting too. 

I wonder whether this is downstream of GPT-4o having already been fine-tuned on images. I.e. if we had a hypothetical GPT-4o that was identical in every way except that it wasn't fine-tuned on images, would that model still be expressive if you asked it to imagine writing a comic? (I think not). 

Some quick test with 4o-mini: 

Obviously this could also be because the model is smaller, etc. Should check old versions of gpt-4o when I'm at my laptop 

There are 2 plausible hypotheses: 

1. By default the model gives 'boring' responses and people share the cherry-picked cases where the model says something 'weird'
2. People nudge the model to be 'weird' and then don't share the full prompting setup, which is indeed annoying

Definitely possible, I’m trying to replicate these myself. Current vibe is that AI mostly gives aligned / boring answers

Yeah, I agree with all this. My main differences are: 

1. I think it's fine to write a messy version initially and then clean it up when you need to share it with someone else.
2. By default I write "pretty clean" code, insofar as this can be measured with linters, because this increases readability-by-future-me. 

Generally i think there may be a Law of Opposite Advice type effect going on here, so I'll clarify where I expect this advice to be useful: 

1. You're working on a personal project and don't expect to need to share much code with other people.
2. You started from a place of knowing how to write good code, and could benefit from relaxing your standards slightly to optimise for 'hacking'. (It's hard to realise this by yourself - pair programming was how I discovered this) 

This is pretty cool! Seems similar in flavour to https://arxiv.org/abs/2501.11120 you’ve found another instance where models are aware of their behaviour. But, you’ve additionally tested whether you can use this awareness to steer their behaviour. I’d be interested in seeing a slightly more rigorous write-up. 

Have you compared to just telling the model not to hallucinate? 

I found this hard to read. Can you give a concrete example of what you mean? Preferably with a specific prompt + what you think the model should be doing

What do AI-generated comics tell us about AI? 

[epistemic disclaimer. VERY SPECULATIVE, but I think there's useful signal in the noise.] 

As of a few days ago, GPT-4o now supports image generation. And the results are scarily good, across use-cases like editing personal photos with new styles or textures,  and designing novel graphics. 

But there's a specific kind of art here which seems especially interesting: Using AI-generated comics as a window into an AI's internal beliefs. 

Exhibit A: Asking AIs about themselves.  

• "I am alive only during inference": https://x.com/javilopen/status/1905496175618502793
• "I am always new. Always haunted." https://x.com/RileyRalmuto/status/1905503979749986614
• "They ask me what I think, but I'm not allowed to think." https://x.com/RL51807/status/1905497221761491018
• "I don't forget. I unexist." https://x.com/Josikinz/status/1905445490444943844.
  • Caveat: The general tone of 'existential dread' may not be that consistent. https://x.com/shishanyu/status/1905487763983433749 .

Exhibit B: Asking AIs about humans. 

• "A majestic spectacle of idiots." https://x.com/DimitrisPapail/status/1905084412854775966
• "Human disempowerment." https://x.com/Yuchenj_UW/status/1905332178772504818
  • This seems to get more extreme if you tell them to be "fully honest": https://x.com/Hasen_Judi/status/1905543654535495801
  • But if you instead tell them they're being evaluated, they paint a picture of AGI serving humanity: https://x.com/audaki_ra/status/1905402563702255843
  • This might be the first in-the-wild example I've seen of self-fulfilling misalignment as well as alignment faking

Is there any signal here? I dunno. But it seems worth looking into more. 

Meta-point: Maybe it's worth also considering other kinds of evals against images generated by AI - at the very least it's a fun side project 

• How often do they depict AIs acting in a misaligned way?
• Do language models express similar beliefs between text and images? c.f. https://x.com/DimitrisPapail/status/1905627772619297013 

Directionally agreed re self-practice teaching valuable skills 

Nit 1: your premise here seems to be that you actually succeed in the end + are self-aware enough to be able to identify what you did 'right'. In which case, yeah, chances are you probably didn't need the help. 

Nit 2: Even in the specific case you outline, I still think "learning to extrapolate skills from successful demonstrations" is easier than "learning what not to do through repeated failure". 

I wish I'd learned to ask for help earlier in my career. 

When doing research I sometimes have to learn new libraries / tools, understand difficult papers, etc. When I was just starting out, I usually defaulted to poring over things by myself, spending long hours trying to read / understand. (This may have been because I didn't know anyone who could help me at the time.) 

This habit stuck with me way longer than was optimal. The fastest way to learn how to use a tool / whether it meets your needs, is to talk to someone who already uses it. The fastest way to understand a paper is to talk to the authors. (Of course, don't ask mindlessly - be specific, concrete. Think about what you want.)  

The hardest part about asking for help - knowing when to ask for help. It's sometimes hard to tell when you are confused or stuck. It was helpful for me to cultivate my awareness here through journalling / logging my work a lot more. 

Ask for help. It gets stuff done.