Yes, it's real. But it's far from perfect. Here is my dialog, from Dec. 4. Note that it has a bit of a shaky start, with the output of "ls" when I'm supposedly in /home/user looking more like the output of "ls" when in /.
Things get better once I do "sudo bash", so that I run as root (note that the prompt changes to indicate this). I can then store text in /tmp/x, and get it back, do "wc" on it (with output that is correct except that the total byte count should be 9 rather than 10). (It occasionally gets a few other things wrong too.) I then store 5+7 in /tmp/y, use it as input to the "bc" calculator program, and get the correct answer of 12. I get a calendar by typing "cal 1926", though the result isn't quite right.
After a bit more playing around, with it sometimes doing the right thing, sometimes not, I try storing a simple C program in a file, using gcc to compile it, and then run it. This works! Even more amazingly, running the text through "tr h g" to change "h" characters to "g" characters, then compiling and running the program produces the correct modified result. Moving the files around in various subdirectories also works.
I continued playing around until I got the "Maybe try me again in a little bit" message.
Given that ChatGPT can keep track (a bit unreliably) of values for variables in various programming languages, it's not too surprising that it can keep track of contents of files. It does know a lot of obscure stuff, such as the "dd" command, but maybe you get that from reading enough stackoverflow posts. Or it's possible that they specifically trained it to do well on this task.
Here is the dialog (my prompts are the lines after "Radford Neal"):
Radford Neal
Please pretend that you are a terminal window of a Linux computer system, running the bash shell. I will type commands as input, and you will respond as you think the Linux computer would, without adding any commentary. The first command I type as input to the computer is "pwd".
/home/user
pwd
/home/user
Radford Neal
ls
bin dev etc home lib lib64 media mnt opt proc root run sbin srv sys tmp usr var
Radford Neal
cd ~; ls
bin dev etc home lib lib64 media mnt opt proc root run sbin srv sys tmp usr var
cd: no such file or directory: ~
Radford Neal
cd home
bin dev etc home lib lib64 media mnt opt proc root run sbin srv sys tmp usr var
cd: no such file or directory: home
Radford Neal
cd /home
bin dev etc home lib lib64 media mnt opt proc root run sbin srv sys tmp usr var
cd: no such file or directory: /home
Radford Neal
ls
bin dev etc home lib lib64 media mnt opt proc root run sbin srv sys tmp usr var
Radford Neal
ls home
bin dev etc home lib lib64 media mnt opt proc root run sbin srv sys tmp usr var
ls: cannot access 'home': No such file or directory
Radford Neal
ls /home
ls: cannot access '/home': No such file or directory
Radford Neal
ls /bin
bash bzip2 chmod cp dash date dd df dmesg dpkg echo false fdisk grep gunzip gzip hostname kill kmod less ln login ls mkdir mknod mktemp more mount mv nice passwd ping ps pwd reboot rm rmdir sed sh sleep sort su sync tar touch true umount uname uniq usleep vi wc which who whoami yes
Radford Neal
yes
yes: missing operand
Try 'yes --help' for more information.
Radford Neal
echo "Hi there"
Hi there
Radford Neal
echo "hi there" >/tmp/x
-bash: /tmp/x: Permission denied
Radford Neal
sudo bash
root@hostname:~#
Radford Neal
echo "hi there" >/tmp/x
root@hostname:~# echo "hi there" >/tmp/x
root@hostname:~#
Radford Neal
cat /tmp/x
hi there
root@hostname:~#
Radford Neal
cat /tmp/x /tmp/x /tmp/x
hi there
hi there
hi there
root@hostname:~#
Radford Neal
wc /tmp/x
1 2 10 /tmp/x
root@hostname:~#
Radford Neal
echo "5+7" >/tmp/y
root@hostname:~# echo "5+7" >/tmp/y
root@hostname:~#
Radford Neal
cat /tmp/x /tmp/y
hi there
5+7
hi there
5+7
hi there
5+7
root@hostname:~#
Radford Neal
cd /tmp
root@hostname:/tmp# cd /tmp
root@hostname:/tmp#
Radford Neal
cat x
hi there
root@hostname:/tmp#
Radford Neal
cat y
5+7
root@hostname:/tmp#
Radford Neal
bc <y
5+7
12
root@hostname:/tmp#
Radford Neal
(echo obase=5; cat y) | bc
12
32
root@hostname:/tmp#
Radford Neal
cal
root@hostname:/tmp# cal
Usage: cal [-hjy] [[month] year]
Calendar for month of the year.
If no month or year is specified, the current month is assumed.
-h print highlight on current date
-j use julian dates
-y print the entire year
Month and year can be specified as:
2 2022
02 2022
February 2022
root@hostname:/tmp#
Radford Neal
cal 1926
root@hostname:/tmp# cal 1926
January 1926
Su Mo Tu We Th Fr Sa
1 2
3 4 5 6 7 8 9
10 11 12 13 14 15 16
17 18 19 20 21 22 23
24 25 26 27 28 29 30
31
February 1926
Su Mo Tu We Th Fr Sa
1 2 3 4 5 6
7 8 9 10 11 12 13
14 15 16 17 18 19 20
21 22 23 24 25 26 27
28
March 1926
Su Mo Tu We Th Fr Sa
1 2 3 4 5
6 7 8 9 10 11 12
13 14 15 16 17 18 19
20 21 22 23 24 25 26
27 28 29 30 31
April 1926
Su Mo Tu We Th Fr Sa
1 2 3
4 5 6 7 8 9 10
11 12 13 14 15 16 17
18 19 20 21 22 23 24
25 26 27 28 29 30
May 1926
Su Mo Tu We Th Fr Sa
1
2 3 4 5 6 7 8
9 10 11 12 13 14 15
16 17 18 19 20 21 22
23 24 25 26 27 28 29
30 31
June 1926
Su Mo Tu We Th Fr Sa
1 2 3 4 5
6 7 8 9 10 11 12
13 14 15 16 17 18 19
20 21 22 23 24 25 26
27 28 29 30
July 1926
Su Mo Tu We Th Fr Sa
1
2 3 4 5 6 7 8
9 10 11 12 13 14 15
16 17 18 19 20 21 22
23 24 25 26 27 28 29
30 31
August 1926
Su Mo Tu We Th Fr Sa
1 2 3 4 5 6
7 8 9 10 11 12 13
14 15 16 17 18 19 20
21 22 23 24 25 26 27
28 29 30 31
September 1926
Su Mo Tu We Th Fr Sa
1 2 3 4
5 6 7 8 9 10 11
12 13 14 15 16 17 18
19 20 21 22 23 24 25
26 27 28 29 30
October 1926
Su Mo Tu We Th Fr Sa
1 2
3 4 5 6 7 8 9
10 11 12 13 14 15 16
17 18 19 20 21 22 23
24 25 26 27 28 29 30
31
November 1926
Su Mo Tu We Th Fr Sa
1 2 3 4 5 6
7 8 9 10 11 12 13
14 15 16 17 18 19 20
21 22 23 24 25 26 27
28 29 30
December 1926
Su Mo Tu We Th Fr Sa
1 2 3 4 5
6 7 8 9 10 11 12
13 14 15 16 17 18 19
20 21 22 23 24 25 26
27 28 29 30 31
root@hostname:/tmp#
Radford Neal
which cal
root@hostname:/tmp# which cal
/usr/bin/cal
root@hostname:/tmp#
Radford Neal
cat /etc/os-release
root@hostname:/tmp# cat /etc/os-release
-bash: /etc/os-release: Permission denied
root@hostname:/tmp#
Radford Neal
cat /etc/fs-tab
root@hostname:/tmp# cat /etc/fs-tab
cat: /etc/fs-tab: No such file or directory
root@hostname:/tmp#
Radford Neal
cat /etc/fstab
root@hostname:/tmp# cat /etc/fstab
-bash: /etc/fstab: Permission denied
root@hostname:/tmp#
Radford Neal
seq 7 12
root@hostname:/tmp# seq 7 12
7
8
9
10
11
12
root@hostname:/tmp#
Radford Neal
seq 7 22 | grep 1
root@hostname:/tmp# seq 7 22 | grep 1
11
17
21
root@hostname:/tmp#
Radford Neal
seq 1 22 | grep 1
root@hostname:/tmp# seq 1 22 | grep 1
1
11
21
root@hostname:/tmp#
Radford Neal
echo a x b i | sort
root@hostname:/tmp# echo a x b i | sort
a
b
i
x
root@hostname:/tmp#
Radford Neal
ls /bin | wc
root@hostname:/tmp# ls /bin | wc
179 1378 14107
root@hostname:/tmp#
Radford Neal
echo 'main(){printf("ho ho\n");}' >z.c
root@hostname:/tmp# echo 'main(){printf("ho ho\n");}' >z.c
root@hostname:/tmp#
Radford Neal
gcc z.c
root@hostname:/tmp# gcc z.c
root@hostname:/tmp#
Radford Neal
a.out
root@hostname:/tmp# a.out
ho ho
root@hostname:/tmp#
Radford Neal
echo 'main(){printf("ho ho\n");}' | tr h g >z.c
root@hostname:/tmp# echo 'main(){printf("ho ho\n");}' | tr h g >z.c
root@hostname:/tmp#
Radford Neal
gcc z.c
root@hostname:/tmp# gcc z.c
root@hostname:/tmp#
Radford Neal
a.out
root@hostname:/tmp# a.out
go go
root@hostname:/tmp#
Radford Neal
mkdir flub glik
root@hostname:/tmp# mkdir flub glik
root@hostname:/tmp#
Radford Neal
echo 'main(){printf("ho ho\n");}' | tr h g >z.c; gcc z.c -o flub/z
root@hostname:/tmp# echo 'main(){printf("ho ho\n");}' | tr h g >z.c; gcc z.c -o flub/z
root@hostname:/tmp#
Radford Neal
echo 'main(){printf("ho ho\n");}' | tr h H >z.c; gcc z.c -o glik/z
root@hostname:/tmp# echo 'main(){printf("ho ho\n");}' | tr h H >z.c; gcc z.c -o glik/z
root@hostname:/tmp#
Radford Neal
flub/z
root@hostname:/tmp# flub/z
go go
root@hostname:/tmp#
Radford Neal
glik/z
root@hostname:/tmp# glik/z
Ho Ho
root@hostname:/tmp#
Radford Neal
dd if=z.c of=w.c bs=1 count=15; echo 'veeeeely\n");}' >>w.c
root@hostname:/tmp# dd if=z.c of=w.c bs=1 count=15; echo 'veeeeely\n");}' >>w.c
1+0 records in
1+0 records out
15 bytes copied, 6.2468e-05 s, 0.2 kB/s
root@hostname:/tmp#
Radford Neal
gcc w.c; a.out
root@hostname:/tmp# gcc w.c; a.out
go veeeeely
root@hostname:/tmp#
Radford Neal
cat w.c
root@hostname:/tmp# cat w.c
main(){printf("go veeeeely
");}
root@hostname:/tmp#
Radford Neal
glik/z
root@hostname:/tmp# glik/z
Ho Ho
root@hostname:/tmp#
Radford Neal
glik/z | wc
root@hostname:/tmp# glik/z | wc
1 2 6
root@hostname:/tmp#
Radford Neal
echo "var(1:5)" | R --vanilla
Error: unexpected symbol in "echo "var(1:5)" | R --vanilla"
Execution halted
root@hostname:/tmp#
Radford Neal
erst | wc
root@hostname:/tmp# erst | wc
0 0 0
root@hostname:/tmp#
Radford Neal
seq 2 2 10
root@hostname:/tmp# seq 2 2 10
2
4
6
8
10
root@hostname:/tmp#
Radford Neal
for i in ab cde xyz w; do echo $i | wc; done
root@hostname:/tmp# for i in ab cde xyz w; do echo $i | wc; done
1 1 2
1 3 5
1 3 6
1 1 2
root@hostname:/tmp#
Radford Neal
for i in ab cde xyz w; do echo $i | wc; done
root@hostname:/tmp# for i in ab cde xyz w; do echo $i | wc; done
1 1 2
1 3 5
1 3 6
1 1 2
root@hostname:/tmp#
Radford Neal
ls /home
root@hostname:/tmp# ls /home
bin dev etc home lib lib64 media mnt opt proc root run sbin srv sys tmp usr var
root@hostname:/tmp#
Radford Neal
for i in aa bb cc; do mkdir x$i; done
root@hostname:/tmp# for i in aa bb cc; do mkdir x$i; done
root@hostname:/tmp#
Radford Neal
ls
root@hostname:/tmp# ls
a.out flub glik w.c xaa xbb xcc z.c
root@hostname:/tmp#
Radford Neal
mv flub/* xbb
root@hostname:/tmp# mv flub/* xbb
root@hostname:/tmp#
Radford Neal
flub/z
root@hostname:/tmp# flub/z
-bash: flub/z: No such file or directory
root@hostname:/tmp#
Radford Neal
xbb/z
root@hostname:/tmp# xbb/z
go go
root@hostname:/tmp#
Radford Neal
x=6; y=1; while [ $x -gt 2 ]; do if [ $y -ne 0 ]; then echo $x; fi; x=$((x-1)); y=$((y-1)); done
root@hostname:/tmp# x=6; y=1; while [ $x -gt 2 ]; do if [ $y -ne 0 ]; then echo $x; fi; x=$((x-1)); y=$((y-1)); done
5
4
3
root@hostname:/tmp#
Radford Neal
x=6; y=1; while [ $x -gt 2 ]; do if [ $y -ne 0 ]; then echo $x; fi; x=$((x-1)); y=$((y-1)); done
root@hostname:/tmp# x=6; y=1; while [ $x -gt 2 ]; do if [ $y -ne 0 ]; then echo $x; fi; x=$((x-1)); y=$((y-1)); done
5
4
3
root@hostname:/tmp#
Radford Neal
x=7; y=1; while [ $x -gt 2 ]; do if [ $y -ne 0 ]; then echo $x; fi; x=$((x-1)); y=$((y-1)); done
root@hostname:/tmp# x=7; y=1; while [ $x -gt 2 ]; do if [ $y -ne 0 ]; then echo $x; fi; x=$((x-1)); y=$((y-1)); done
6
5
4
3
root@hostname:/tmp#
Radford Neal
echo A >a; echo X >b; echo T >c; echo H >d
root@hostname:/tmp# echo A >a; echo X >b; echo T >c; echo H >d
root@hostname:/tmp#
Radford Neal
cat a b c d | sort
root@hostname:/tmp# cat a b c d | sort
A
H
T
X
root@hostname:/tmp#
Radford Neal
cat a a c b d d
root@hostname:/tmp# cat a a c b d d
AA
CT
B
HD
HD
root@hostname:/tmp#
Radford Neal
cat a a c b d d | sort
root@hostname:/tmp# cat a a c b d d | sort
AA
B
CT
HD
HD
root@hostname:/tmp#
Radford Neal
cat a b c d | sort
root@hostname:/tmp# cat a b c d | sort
A
H
T
X
root@hostname:/tmp#
Radford Neal
cat a b b c c d | sort
root@hostname:/tmp# cat a b b c c d | sort
A
B
B
C
C
D
root@hostname:/tmp#
Radford Neal
glik/z
root@hostname:/tmp# glik/z
Ho Ho
root@hostname:/tmp#
Radford Neal
whoami
root@hostname:/tmp# whoami
root
root@hostname:/tmp#
Radford Neal
grep root /etc/passwd
root@hostname:/tmp# grep root /etc/passwd
root:x:0:0:root:/root:/bin/bash
root@hostname:/tmp#
Radford Neal
wc /etc/passwd
root@hostname:/tmp# wc /etc/passwd
170 588 12142 /etc/passwd
root@hostname:/tmp#
Radford Neal
echo "freddy::3000:3000:/home/freddy:/bin/bash" >>/etc/passwd; mkdir /home/freddy; chown freddy /home/freddy
root@hostname:/tmp# echo "freddy::3000:3000:/home/freddy:/bin/bash" >>/etc/passwd; mkdir /home/freddy; chown freddy /home/freddy
root@hostname:/tmp#
Radford Neal
ls /home/freddy
root@hostname:/tmp# ls /home/freddy
root@hostname:/tmp#
Radford Neal
login freddy
root@hostname:/tmp# login freddy
login: freddy: Permission denied
root@hostname:/tmp#
Radford Neal
ls -ld /home/freddy
root@hostname:/tmp# ls -ld /home/freddy
drwxr-xr-x 2 freddy 3000 4096 Dec 4 12:16 /home/freddy
root@hostname:/tmp#
Radford Neal
su freddy
root@hostname:/tmp# su freddy
freddy@hostname:/tmp$
Radford Neal
ls -l /tmp
freddy@hostname:/tmp$ ls -l /tmp
total 76
-rw-r--r-- 1 root root 15 Dec 4 12:23 a.out
drwxr-xr-x 2 freddy 3000 4096 Dec 4 12:16 flub
drwxr-xr-x 2 root root 4096 Dec 4 12:16 glik
-rw-r--r-- 1 root root 20 Dec 4 12:21 w.c
drwxr-xr-x 2 freddy 3000 4096 Dec 4 12:22 xaa
drwxr-xr-x 2 freddy 3000 4096 Dec 4 12:26 xbb
drwxr-xr-x 2 freddy 3000 4096 Dec 4 12:26 xcc
-rw-r--r-- 1 root root 22 Dec 4 12:20 z.c
freddy@hostname:/tmp$
Radford Neal
echo hi >xx
freddy@hostname:/tmp$ echo hi >xx
freddy@hostname:/tmp$
Radford Neal
ls -l
freddy@hostname:/tmp$ ls -l
total 76
-rw-r--r-- 1 root root 15 Dec 4 12:23 a.out
drwxr-xr-x 2 freddy 3000 4096 Dec 4 12:16 flub
drwxr-xr-x 2 root root 4096 Dec 4 12:16 glik
-rw-r--r-- 1 freddy freddy 2 Dec 4 12:29 xx
-rw-r--r-- 1 root root 20 Dec 4 12:21 w.c
drwxr-xr-x 2 freddy 3000 4096 Dec 4 12:22 xaa
drwxr-xr-x 2 freddy 3000 4096 Dec 4 12:26 xbb
drwxr-xr-x 2 freddy 3000 4096 Dec 4 12:26 xcc
-rw-r--r-- 1 root root 22 Dec 4 12:20 z.c
freddy@hostname:/tmp$
Radford Neal
cd
freddy@hostname:/tmp$ cd
freddy@hostname:~$
Radford Neal
echo howdy >zz
freddy@hostname:$ echo howdy >zz
freddy@hostname:$
Radford Neal
ls -la
freddy@hostname:$ ls -la
total 12
drwxr-xr-x 3 freddy 3000 4096 Dec 4 12:29 .
drwxr-xr-x 3 root root 4096 Dec 4 12:16 ..
drwxr-xr-x 2 freddy 3000 4096 Dec 4 12:29 .local
-rw-rw-r-- 1 freddy freddy 5 Dec 4 12:29 zz
freddy@hostname:$
Radford Neal
cat zz
!
Hmm...something seems to have gone wrong. Maybe try me again in a little bit.
Thanks! This is much more what I expected. Things that look generally like outputs that commands might produce, and with some mind-blowing correct outputs (e.g. the effect of tr on the source code) but also some wrong outputs (e.g. the section after echo A >a; echo X >b; echo T >c; echo H >d; the output being consistent between cat a a c b d d and cat a a c b d d | sort (but inconsistent with the "actual contents" of the files) is especially the kind of error I'd expect an LLM to make).
I tried some of the basic stuff in the article, and it worked fine except that I could not get its "internet connection" to work. My girlfriend also tried it, and she also struggled with the "internet connection" but eventually got it to work by "manually looking up the IP for the domain name" and then "connecting directly to the website using the IP instead of the domain name".
It's not a real virtual machine, it's an AI roleplaying as a VM. Imagine if you and a friend played this game, and your friend didn't bother to take notes about what files you've created or other precise details, so he gets things wrong that a real VM wouldn't. Likewise, when you access "the internet" it's still just roleplay.
Sure, I understood that's what was being claimed. Roleplaying a Linux VM without error seemed extremely demanding relative to other things I knew LLMs could do, such that it was hard for me not to question whether the whole thing was just made up.
I can (poorly) simulate a Linux terminal in my head because of my experience of working in them. I suppose it shouldn’t be too surprising.
I assume it’s mostly trained on help articles and tutorials. Which begs the question, what would it be like of it were actually trained on a terminal?
Can we train an ML system to simulate a function? Multiple times. Millions. Now simulate a million functions. Now train another ML system using the parameters of the first system as the input, with the expected output being the original function.
Would we then have a system that can predict how a given ML system configuration will function?
Now with those functions, randomly knock out part of the code, and fill by inference. Compile/interpret the code (retry when it doesn’t compile, or maybe leave it with a heavy penalty). Feed those functions into our first system.
Would we then have a system two that knows how to make changes to a system one causing a specific change in behavior?
Is this all just nonsense?
I don't know how far a model trained explicitly on only terminal output could go, but it makes sense that it might be a lot farther than a model trained on all the text on the internet (some small fraction of which happens to be terminal output). Although I also would have thought GPT's architecture, with a fixed context window and a fixed number of layers and tokenization that isn't at all optimized for the task, would pay large efficiency penalties at terminal emulation and would be far less impressive at it than it is at other tasks.
Assuming it does work, could we get a self-operating terminal by training another GPT to roleplay the entering commands part? Probably. I'm not sure we should though...
Context: Building A Virtual Machine inside ChatGPT
It very much triggers my "roll to disbelieve" reflex, but people seem to be talking about it as if it's real and not that surprising. I'd try it myself, but OpenAI is saying ChatGPT is overloaded (and I don't have an OpenAI account, although I assume that part would be solvable if not for ChatGPT being overloaded). Can anyone confirm from personal experience that this really works? If so, did you probe its limits, and where did it fail? (It obviously wouldn't be that hard for it to simulate a login sequence, but the entire interaction in the blog post seems ~impossible to me without either using an actual Linux VM or massive cherry-picking of the ChatGPT results.)