ChatGPT Gets "Absolutely Wrecked" By An Atari Video Chess Game Built In 1979

OpenAI's ChatGPT, though it still continues to make up plausible-sounding nonsense when it is fresh out of factual information, remains an impressive tool. Despite concerns over copyright, the model (and others) has impressed users with its ability to generate roughly the text you are looking for (albeit, again, likely with a few factual errors).

Researchers have tested its limits, finding, for example, that it is surprisingly good at piloting spacecraft. But there are plenty of things it still struggles with. Despite improvements, it is still not very good at math.

In a new blow to the feeling-less large language model, it appears to be worse than an Atari chess game first made in the 1970s. It is worse at chess than a 46-year-old computer game.

In a short (and not particularly scientific) experiment, Citrix engineer Robert Jr. Caruso decided to pit ChatGPT against the Atari 2600’s chess engine "and see what happens". 

"This was after a conversation we had regarding the history of AI in Chess which led to it volunteering to play Atari Chess," Caruso explained in a LinkedIn post. "It wanted to find out how quickly it could beat a game that only thinks 1-2 moves ahead on a 1.19 MHz CPU."

While ChatGPT is one of the most advanced artificial intelligence (AI) models out there, with nearly half a century of tech improvements behind it, it was not designed to play chess specifically, making the outcome unclear.

Despite ChatGPT's characteristic confidence, it did not go well, with Caruso having to correct the machine's board awareness several times per turn, and advise it not to make terrible moves.

"ChatGPT got absolutely wrecked on the beginner level," Caruso explained. "Despite being given a baseline board layout to identify pieces, ChatGPT confused rooks for bishops, missed pawn forks, and repeatedly lost track of where pieces were – first blaming the Atari icons as too abstract to recognize, then faring no better even after switching to standard chess notation. It made enough blunders to get laughed out of a third grade chess club."

After several times of promising it would improve if it could just start over, ChatGPT conceded to the Atari chess game. While that might not be too surprising – the chess game was designed to play chess, after all – it highlights that we've still got some way to go before general purpose AIs can outdo their purpose-specific counterparts. Even when that counterpart was built in 1979, and runs on a 1.19 MHz CPU.