I know it has been a while, but I came across a study in my news feed today and remembered this conversation, so I'm sharing the link.
ChatGPT as a cognitive crutch: Evidence from a randomized controlled trial on knowledge retention
https://www.sciencedirect.com/science/article/pii/S2590291125010186
Cool, thanks for sharing.
reading through it id like to bring up some points I find interesting.
The chatgpt spent 45% less time studying than those doing it traditionally and scored 11% lower on their retention test (57% vs 68%)
In one section they discuss that the biggest disparity happens on complex subjects, it seems a structured course is better. This makes the next part I mention below an interesting choice to me.
From what I read the content for studying was "AI/ML topics spanning foundations, methods, applications, and ethics". This seems to be a pretty complicated subject to me. The students chosen for this test are Business Administration students. Im guessing its a pretty large learning curve from that to machine learning.
They do mention that there is still disparity on other topics but it looked like it was about half as bad.
In my personal opinion I do think it is a crutch, a stiffer crutch than google even, however I think they may have painted the picture a little worse than it is.
There is a lot of things in this world to know about or understand, sometimes knowing where to find an answer is our best option.
This discussion does remind me of math class at least when I was in school, simply putting the correct answer on the test did not give you full points, you had to write the process you used to get to the answer, demonstrating you actually understand the process. If you truly want to learn something effort, time, involvement, execution all greatly help in retention. For me putting it into real world practice is most beneficial.
This is just the latest study I came across. I've seen others. Most often they are on students and how they become dependent on AI instead of learning to think for themselves.
Knowing how to use tools to find facts is often more important than remembering every little detail, but critical thinking and problem solving are vital skills on their own that go beyond knowing specific facts. The problem with comparing AI to using a search engine to find facts, is that Generative AI/LLM is not a tool for finding facts. It is a tool that makes up answers based on how it was trained. Sometimes it will give you facts. Sometimes it makes up completely random answers and you have to know enough of the topic to be able to know the difference, in which case you probably could have solved the problem yourself.
In cases of programming, many people are asking it to create new things, be it art or code. That is no longer working with simple facts but having it do the work of solving the problem for you. If people, especially students learning new things, rely on AI to do the creative thinking and problem solving for them, then they will not develop the neural pathways to do it themselves. Their brains will become wired to always go to the AI tools for the answer. It will be a self-feeding cycle of dependence.
We live in stressful times, and people are often looking for shortcuts. It is easy to start by only using something occasionally for certain situations. When you start to feel that it works well enough and is convenient , you may start to decide to use it more and more. Soon, it may be the first place you look whenever you run into something you don't have the immediate answer for. People will do a lot for convenience. The more you use it, then more you become dependent on it. That is the trend that I see being reported as the big problem with the way Gen AI is being used these days. People stop being able to think for themselves because having the AI do all the thinking when they run into a challenge is quicker and easier, and life is stressful enough that having it take care of these things is so convenient. The more you use it, the more you are likely to trust it. You may start by using it as one of many sources, and even validating everything. The more it works, the more confidence you will have in it. Over time, as your confidence grows, you are more likely to do less validating and comparing to other sources. That is a very common thing for people to do with repetition of things that work for them. That kind of complacency is how many mistakes and accidents happen, not just specific to AI. One problem here is that Gen AI is constantly changing, and something that works today may not tomorrow, but you may not know what changed or when.
In addition to all of the other problems with using Gen AI, it is absolutely not at the point where it should simply be trusted to give you the right answer all the time. It should at best be seen as giving you a starting point that must be validated by someone familiar with the topic. It is the kind of thing that is good at taking a huge set of data and reducing it down to the most likely candidates, which can then be handled by people, but should not be deciding on the final answer. Far too many people try it a few times and think it has all the answers and simply trust whatever it says. I've heard people in business environment praise how amazing it is and talk about putting it everywhere, without any need for validation, because it worked so well for them in the few personal things they tried. Someday AI may be able to do more, but in its current state it is being marketed as more than it is, and people are relying on it too much.
