I wouldn’t either but that’s exactly what lmsys.org found.

That blog post had ratings between 858 and 1169. Those are slightly higher than the average rating of human users on popular chess sites. Their latest leaderboard shows them doing even better.

https://lmarena.ai/leaderboard has one of the Gemini models with a rating of 1470. That’s pretty good.

I imagine the “author” did something like, “Search http://google.scholar.com/ find a publication where AI failed at something and write a paragraph about it.”

It’s not even as bad as the article claims.

Atari isn’t great at chess. https://chess.stackexchange.com/questions/24952/how-strong-is-each-level-of-atari-2600s-video-chess
Random LLMs were nearly as good 2 years ago. https://lmsys.org/blog/2023-05-03-arena/
LLMs that are actually trained for chess have done much better. https://arxiv.org/abs/2501.17186

Like humans are way better at answering stuff when it’s a collaboration of more than one person. I suspect the same is true of LLMs.

It is.

It’s really common for non-language implementations of neural networks. If you have an NN that’s right some percentage of the time, you can often run it through a bunch of copies of the NNs and take the average and that average is correct a higher percentage of the time.

Aider is an open source AI coding assistant that lets you use one model to plan the coding and a second one to do the actual coding. It works better than doing it in a single pass, even if you assign the the same model to planing and coding.

The Israelis regularly murder journalists and civilians. The danger was quite real.

Sometimes it seems like most of these AI articles are written by AIs with bad prompts.

Human journalists would hopefully do a little research. A quick search would reveal that researches have been publishing about this for over a year so there’s no need to sensationalize it. Perhaps the human journalist could have spent a little time talking about why LLMs are bad at chess and how researchers are approaching the problem.

LLMs on the other hand, are very good at producing clickbait articles with low information content.

That makes sense. Not everything needs to be testable. There are many interesting and important ideas outside of science.

The main problem would be if someone wanted to set policy based on it. That includes the implicit experiment of, “If we adopt policy A we can expect outcome B.” If we haven’t tested that before turning it into a policy, the policy itself becomes the experiment, and then we need to be very careful about the ethics surrounding such an experiment.

It’s kind of like string theory. It has a bunch of interesting conjectures but nobody can figure out a way to test any of it.

Take the “selfish gene” (the idea predates Dawkins). One of the theories states that it may be evolutionarily advantageous for an individual to sacrifice themselves for the group if they share enough DNA. They lose the DNA in their bodies but save the exact same DNA in the bodies of their extended family. That’s a nice idea and you can get the math to work out in game theory models but how do we test if that’s why ducks sometimes lag behind when a hunter tries to shoot them?

That’s not to say it can never be tested. There are other cases where we needed to wait for technological breakthroughs until theories could actually be tested.

Glock is Austrian. They just love the US market.