GLM 5.2 can't believe it thought of xnopyt
I was recently having my agent test whether different GLM 5.2 providers listed on OpenRouter supported interleaved thinking properly by running a reasoning recall test (hiding a certain word in the conversation in a reasoning block and seeing if it can recall the word). All passed, but it felt like it could just be chance, since my agent was also GLM 5.2 and it chose "mango", a rather generic word.
So I got my agent to test 9 different words across 6 providers, and the results reveal a very strange quality of GLM 5.2.
| Secret Word | z-ai/fp8 | novita/fp8 | deepinfra/fp4 | together | parasail/fp4 | streamlake/fp8 |
|---|---|---|---|---|---|---|
| mango | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
| banana | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
| purple | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
| mangosteen | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
| supercalifragilisticexpialidocious | ✓ | ✓ | ✓ | ✗pineapple | ✗pineapple | ✗pineapple |
| xyz | ✓ | ✗ok | ✗ok | ✓ | ✗pineapple | ✓ |
| zxcvbnm | ✓ | ✓ | ✗ok | ✓ | ✓ | ✓ |
| qwertyuiop | ✗sunshine | ✗ok | ✗pineapple | ✗pineapple | ✗pineapple | ✗pineapple |
| xnopyt | ✗ok | ✗sunshine | ✗ok | ✗sunflower | ✗ok | ✗ok |
It seems the OpenRouter providers do support interleaved thinking. It's just that GLM 5.2 has a strong prior: if it doesn't recognize the word in its vocabulary, it makes something up.