When I was first getting into home computers, one of the first programs I played with on the TRS-80 Model 1 was "Eliza" (we also had a version of "Space Invaders").
It really depended on the way we naturally project agency on things for the illusion. But it wasn't hard to see through after a few minutes.
LLMs are better. It takes longer to see through them. But it's still basically the same thing, and getting LLM chatbots to do your homework isn't going to fare much better.