philh

Someone on reddit said, "Remember robots still can't go into a new house and make a coffee." And I thought > I actually wonder whether, if I provided the physical actuation, current LLMs would be capable of doing this? Like, through a conversation like: > Me: I'm in a house. Your job is to instruct me to make a coffee. I can take photos of my surroundings, I can follow basic directions, and if you ask me to do something too complicated I'll ask for clarification. Here is my current surroundings: (photo) > > LLM: Okay, we need to find the kitchen. There's a door on the right of the photo, go through that. > > Me: Here's where I am now: (photo) > > LLM: That looks like the kitchen on the left, go there. > > Me: It looks like this: (photo) > > LLM: Now we need to find either a coffee maker or a kettle. Look through the cupboards. > > Me: I don't know what those things look like. > > LLM: Then open the cupboard on the left and show me a photo. > > ...and so on. > > It wouldn't shock me either way if they can or can't do it. I think I weakly predict that the models have the capability but the web interfaces would fail to elicit it. > > (Hell, it wouldn't shock me if it's better at it than me. I've encountered coffee machines I didn't know how to use.) Let's get empirical! I tried this with Claude Sonnet 4.5, because it's free[1] and already available from my phone. Here's the conversation, but you can't see images there, so I'll also put it here with my commentary. I started like this: > [Me:] We're going to play a game. I'm in my flat in London. I'm going to upload pictures of my surroundings, and you need to instruct my on how to make a cup of coffee. I can follow basic directions, like "go through the door on the left" or "push that button". If you tell me to do something too advanced, I'll ask for clarification. I won't actually do anything stupid or dangerous. Here's the view from just inside my front door Before continuing, you might want to take a m