For the first experiment with 354 participants, one group was tasked with solving 15 basic fraction problems without help, while another had ChatGPT open on the side to use for assistance and to even ask for the answer. The group with AI access started off more accurate than the group without it. Then, after 12 problems, researchers took away the AI tool. Almost immediately, the people in that group stopped solving questions accurately.
A similar thing happened when they ran the experiment on a second, larger group of 667 participants. Whereas the AI users got answers wrong or gave up entirely when the AI assistance was withdrawn, the group without AI stuck with the task and were more successful at the end.
Researchers then ran a third experiment with 201 participants and an SAT reading comprehension prompt. The results echoed those from the fraction problems: When the AI tool was removed, persistence and accuracy dropped.
I wonder if this kind of test would show the same results if you compared people using calculators for math problems vs. those without. From the basic operations to doing trig and calculus, if you provide an easier way they’re going to take it, and unfortunately not going to pick up on how the answer is obtained, which an LLM will usually tell in the process. Doing something yourself always teaches you better than watching it, or just being told the answer.
If the tests are implying or showing that people who already were proficient in doing these tests and would have done well without started declining after exposure to an LLM’s help, and that persisted… that’s a different problem and one that should be looked into. Losing skills you had because of help? That shouldn’t happen.
A lot of pro AI people like to compare LLMs to calculators but it’s not comparable at all unless the math you’re doing is simple arithmetic. If you’re doing calculus, the skill you’re working on isn’t arithmetic, so the calculator isn’t much of a factor. Lots of people forget how to do long division after using calculators on the other hand.
Losing skills you had because of help? That shouldn’t happen.
That’s exactly what should happen. Cognitive thinking skills are use it or lose it. If you use an LLM to replace your brain, your brain will get worse.
Tech evangelists rely heavily on logical fallacies. False equivalence is one of the main ones. “A thing happened before therefore this tech is nothing to worry about”. They’re redirecting the argument to bait people to try to prove why calculators are bad. But of course the past already happened and nobody is going prove otherwise.
but it’s not comparable at all unless the math you’re doing is simple arithmetic.
That’s what they were studying on the math side
For the first experiment with 354 participants, one group was tasked with solving 15 basic fraction problems
Also
If you’re doing calculus, the skill you’re working on isn’t arithmetic, so the calculator isn’t much of a factor.
Calculators would be a big factor in solving calculus problems. Haven’t done calculus in a while but if I recall correctly you are breaking down or rearranging the problem into smaller steps that are more basic arithmetic. One part of a calculus problem could be finding out what the square root of 7 is , if you have a calculator to do that you’re more likely to solve the problem more accurately and to not give up on the problem, which were the two metrics this study was looking at.
Realistically, you would just write root 7 as a coefficient of your answer if you’re performing this math long hand. No one really expects anyone to be able to calculate things like that.
Yep, especially in mathematics a calculator is not much of a help in calculus. The moment you put numbers in whatever thing you wanna solve is the point where the hard part is already done.
Over time yes, but this says skills were lost during a single session or short term. It was a random group of people, so unknown on what skills they did have before. I think what they observed wasn’t a permanent loss, but a placebo effect of having extra help and it was taken away, so for the rest of the test they felt lacking.
My example of calculators was implying one that could do the test work. All calculators have trig functions, and some have calculus abilities. Would a student forget how to find cosine if the calculator was taken away, or do a derivative of a function? Not in one test. But if they were using it first half and then had to do it themselves for the rest, I imagine there’s some psychological stuff going on that makes you feel like you’ve lost that skill.
Doing something yourself always teaches you better than watching it, or just being told the answer.
A neat result from pedagogical studies is that being given the worked solutions to new problems to study from is much more effective for learning the material than simply practicing.
Students who were assigned practice problems performed worse on later tests than those given the same problems and all of the worked-out solutions for them. The explanation from the research was - when learning a new topic, immediate practice isn’t beneficial because the student doesn’t yet know what to practice. If they are lost when trying the problem, they can easily end up practicing the wrong skills by not knowing where to start. Those with the solutions have some of the cognitive burden removed, and are better able to internalize the task and generalize it to later problems.
I completely agree with you on LLMs and I think they’ll be a disaster for education. But I’ve always found it interesting that being given the answer can be useful for learning in certain circumstances.
A neat result from pedagogical studies is that being given the worked solutions to new problems to study from is much more effective for learning the material than simply practicing.
Finally, something to justify my still-existent hatred of my middle school geometry teacher for literally telling us ‘the theorems are on the board’ any time we asked her for help trying to solve a problem or for her to show us an example problem. Why the fuck were you even teaching.
That was one of my gripes with my experience in college vs hs. In my hs at least, we were taught the most difficult combination of a problem and then everything that we practiced would be able to be worked out from there. In college, we were often taught the simplest example and somehow expected to derive the most difficult version of it on the problem sets. That’s how you get students who study really hard for an exam and half of them fail. They never taught the proper way to solve the problem!
I was out of college by the time LLMs came out but I still believe they could be helpful in this scenario if fine tuned and limited as a tutor not an answer-giver. Obviously if you have access to a good human tutor (who doesn’t call you dumb like my friend had at the time), but I didn’t have access to one.
There are different ways to learn, some work better for certain people than others.
I would think that the best way to learn is neither of these options, but to be taught how to do the work by someone who understands (like a teacher) and then made to do it after it has been thoroughly explained to you and practiced with you.
That’s what works best for me anyhow, that or literally reading the textbook, lol, but that’s less efficient time-wise.
Immediate practice on something you haven’t been walked through yet? I guess the way I phrased it made it sound like jumping in yourself with no guide is better. You definitely need to be taught first, or at least as you go, and not figure it out on your own. The point was that simply watching someone do something isn’t enough, you have to develop the process of getting to that point yourself. But only after seeing that route, otherwise you will take lots of wrong paths.
Yes, I’m aware of what they tested and what fractions are…
I was just expanding the point to other things that are comparable. And an LLM for a reading comprehension test is exactly what a large language model would be best suited for, so in essence a language “calculator”.
Doesn’t change the rest of the discussion, which was about the brain’s ability to jump back into things when the crutch is taken away.
I don’t think it’s fair to say that fractions are similar to trigonometry and calculus aside from the fact that they’re all types of math
Very few people do trigonometry or calculus in their heads, I’m sure most people who know it really well could do a cosine on paper, but that’s a much higher level operation than a fraction
Um, you know all of those were developed before calculators, right?
I learned both trig and calc on the principles in how they were derived, so yes, when I was learning and using them, I could do some of it in my head or with a few figures on paper.
But I never said “similar” at all, I was using them as comparable subjects that could be done by hand or with aid.
You’re so obsessed with calling out someone on not mentioning fractions you’ve missed the point of the test, article, and this thread. They use fractions because it was above simple math but not so high level that some of the random people picked might not know it. Which going back to the real discussion, having randoms whose skill isn’t known even halfway through the test might have been a flaw in the test.
Um, you know all of those were developed before calculators, right?
What does the timing of the advent of the calculator have to do with what I said?
But I never said “similar” at all, I was using them as comparable subjects
I would contest that they are comparable subjects. Entry level piano lessons and music theory are both technically of the same subject, but wildly different in practice. One is suitable for beginners or lay people, the other is not.
You’re so obsessed with calling out someone on not mentioning fractions you’ve missed the point of the test, article, and this thread.
You are projecting your own obsession of using the poor comparison of “AI = Calculator” onto me. That’s why I keep bringing up fractions, it is an easy-to-understand (for most) rebuttal to your logically fallacious proposal.
They use fractions because it was above simple math but not so high level that some of the random people picked might not know it.
Exactly, that’s why your comparison doesn’t work.
Which going back to the real discussion, having randoms whose skill isn’t known even halfway through the test might have been a flaw in the test.
What changes would you make to the experiment to remove the flaw you’re perceiving?
How do you expect that change would impact the outcome?
As I’ve said before, the abilities of the ones with ChatGPT assist don’t seem to be known, as they started off with help. The assumption is that they might on average be the same as the others, but we don’t know that. A simple fix to start is to offer the LLM help in the middle of the test for them, see how the curve of ability goes up and down, and compare before and after.
Not answering the rest, I was more interested in the study and what it was claiming and not arguing about missing context of commentary. I think the study may be right in suggesting people today have more trouble understanding things than they used to, but it’s far deeper and older than LLMs. They are just the latest part of the bigger problem.
I don’t think that change would negatively impact the experiment, but for what they were trying to measure I’m unsure of how necessary that step would be.
They weren’t measuring how smart or good at math the participants were, they were specifically measuring changes in focus and persistence (so basically the amount of effort put forward). The findings show that people tend to just give up when you take the LLM away, and I think that with your proposed change you’d still get largely the same findings.
I would agree that LLMs aren’t the entire issue, but they are making the issue far worse at an exponential rate.
I’d be interested to read a study about what factors and/or traits predispose someone to trust an LLM, or allow themselves to become dependent on it.
I wonder if this kind of test would show the same results if you compared people using calculators for math problems vs. those without. From the basic operations to doing trig and calculus, if you provide an easier way they’re going to take it, and unfortunately not going to pick up on how the answer is obtained, which an LLM will usually tell in the process. Doing something yourself always teaches you better than watching it, or just being told the answer.
If the tests are implying or showing that people who already were proficient in doing these tests and would have done well without started declining after exposure to an LLM’s help, and that persisted… that’s a different problem and one that should be looked into. Losing skills you had because of help? That shouldn’t happen.
A lot of pro AI people like to compare LLMs to calculators but it’s not comparable at all unless the math you’re doing is simple arithmetic. If you’re doing calculus, the skill you’re working on isn’t arithmetic, so the calculator isn’t much of a factor. Lots of people forget how to do long division after using calculators on the other hand.
That’s exactly what should happen. Cognitive thinking skills are use it or lose it. If you use an LLM to replace your brain, your brain will get worse.
Tech evangelists rely heavily on logical fallacies. False equivalence is one of the main ones. “A thing happened before therefore this tech is nothing to worry about”. They’re redirecting the argument to bait people to try to prove why calculators are bad. But of course the past already happened and nobody is going prove otherwise.
That’s what they were studying on the math side
Also
Calculators would be a big factor in solving calculus problems. Haven’t done calculus in a while but if I recall correctly you are breaking down or rearranging the problem into smaller steps that are more basic arithmetic. One part of a calculus problem could be finding out what the square root of 7 is , if you have a calculator to do that you’re more likely to solve the problem more accurately and to not give up on the problem, which were the two metrics this study was looking at.
Realistically, you would just write root 7 as a coefficient of your answer if you’re performing this math long hand. No one really expects anyone to be able to calculate things like that.
Yep, especially in mathematics a calculator is not much of a help in calculus. The moment you put numbers in whatever thing you wanna solve is the point where the hard part is already done.
Over time yes, but this says skills were lost during a single session or short term. It was a random group of people, so unknown on what skills they did have before. I think what they observed wasn’t a permanent loss, but a placebo effect of having extra help and it was taken away, so for the rest of the test they felt lacking.
My example of calculators was implying one that could do the test work. All calculators have trig functions, and some have calculus abilities. Would a student forget how to find cosine if the calculator was taken away, or do a derivative of a function? Not in one test. But if they were using it first half and then had to do it themselves for the rest, I imagine there’s some psychological stuff going on that makes you feel like you’ve lost that skill.
They were tested on fractions
Did you use an LLM to summarize the study for you or something?
And reading comprehension.
We’re discussing the effects of the test, not what the test was.
The content of the test is relevant to the outcome
A neat result from pedagogical studies is that being given the worked solutions to new problems to study from is much more effective for learning the material than simply practicing.
Students who were assigned practice problems performed worse on later tests than those given the same problems and all of the worked-out solutions for them. The explanation from the research was - when learning a new topic, immediate practice isn’t beneficial because the student doesn’t yet know what to practice. If they are lost when trying the problem, they can easily end up practicing the wrong skills by not knowing where to start. Those with the solutions have some of the cognitive burden removed, and are better able to internalize the task and generalize it to later problems.
I completely agree with you on LLMs and I think they’ll be a disaster for education. But I’ve always found it interesting that being given the answer can be useful for learning in certain circumstances.
Finally, something to justify my still-existent hatred of my middle school geometry teacher for literally telling us ‘the theorems are on the board’ any time we asked her for help trying to solve a problem or for her to show us an example problem. Why the fuck were you even teaching.
That was one of my gripes with my experience in college vs hs. In my hs at least, we were taught the most difficult combination of a problem and then everything that we practiced would be able to be worked out from there. In college, we were often taught the simplest example and somehow expected to derive the most difficult version of it on the problem sets. That’s how you get students who study really hard for an exam and half of them fail. They never taught the proper way to solve the problem!
I was out of college by the time LLMs came out but I still believe they could be helpful in this scenario if fine tuned and limited as a tutor not an answer-giver. Obviously if you have access to a good human tutor (who doesn’t call you dumb like my friend had at the time), but I didn’t have access to one.
There are different ways to learn, some work better for certain people than others.
I would think that the best way to learn is neither of these options, but to be taught how to do the work by someone who understands (like a teacher) and then made to do it after it has been thoroughly explained to you and practiced with you.
That’s what works best for me anyhow, that or literally reading the textbook, lol, but that’s less efficient time-wise.
Immediate practice on something you haven’t been walked through yet? I guess the way I phrased it made it sound like jumping in yourself with no guide is better. You definitely need to be taught first, or at least as you go, and not figure it out on your own. The point was that simply watching someone do something isn’t enough, you have to develop the process of getting to that point yourself. But only after seeing that route, otherwise you will take lots of wrong paths.
Calculators don’t help with reading comprehension (3rd Test)
Also, they’re not talking about trig and calculus, they’re talking about fractions
You know (hopefully), like 1/5 =0.2? Or 6/18=1/3=0.333?
Yes, I’m aware of what they tested and what fractions are…
I was just expanding the point to other things that are comparable. And an LLM for a reading comprehension test is exactly what a large language model would be best suited for, so in essence a language “calculator”.
Doesn’t change the rest of the discussion, which was about the brain’s ability to jump back into things when the crutch is taken away.
I don’t think it’s fair to say that fractions are similar to trigonometry and calculus aside from the fact that they’re all types of math
Very few people do trigonometry or calculus in their heads, I’m sure most people who know it really well could do a cosine on paper, but that’s a much higher level operation than a fraction
Um, you know all of those were developed before calculators, right?
I learned both trig and calc on the principles in how they were derived, so yes, when I was learning and using them, I could do some of it in my head or with a few figures on paper.
But I never said “similar” at all, I was using them as comparable subjects that could be done by hand or with aid.
You’re so obsessed with calling out someone on not mentioning fractions you’ve missed the point of the test, article, and this thread. They use fractions because it was above simple math but not so high level that some of the random people picked might not know it. Which going back to the real discussion, having randoms whose skill isn’t known even halfway through the test might have been a flaw in the test.
What does the timing of the advent of the calculator have to do with what I said?
I would contest that they are comparable subjects. Entry level piano lessons and music theory are both technically of the same subject, but wildly different in practice. One is suitable for beginners or lay people, the other is not.
You are projecting your own obsession of using the poor comparison of “AI = Calculator” onto me. That’s why I keep bringing up fractions, it is an easy-to-understand (for most) rebuttal to your logically fallacious proposal.
Exactly, that’s why your comparison doesn’t work.
What changes would you make to the experiment to remove the flaw you’re perceiving?
How do you expect that change would impact the outcome?
As I’ve said before, the abilities of the ones with ChatGPT assist don’t seem to be known, as they started off with help. The assumption is that they might on average be the same as the others, but we don’t know that. A simple fix to start is to offer the LLM help in the middle of the test for them, see how the curve of ability goes up and down, and compare before and after.
Not answering the rest, I was more interested in the study and what it was claiming and not arguing about missing context of commentary. I think the study may be right in suggesting people today have more trouble understanding things than they used to, but it’s far deeper and older than LLMs. They are just the latest part of the bigger problem.
I don’t think that change would negatively impact the experiment, but for what they were trying to measure I’m unsure of how necessary that step would be.
They weren’t measuring how smart or good at math the participants were, they were specifically measuring changes in focus and persistence (so basically the amount of effort put forward). The findings show that people tend to just give up when you take the LLM away, and I think that with your proposed change you’d still get largely the same findings.
I would agree that LLMs aren’t the entire issue, but they are making the issue far worse at an exponential rate.
I’d be interested to read a study about what factors and/or traits predispose someone to trust an LLM, or allow themselves to become dependent on it.
Some kind of comparison group would have made the result more informative.
Replace this with math and calculators. Similar results? Then it’s not AI’s fault, but a known phenomenon.
Of course, they would never even get the funding for this “experiment” if they didn’t have AI fearmongering in the title.
You can’t do fractions without a calculator?
I assume you’re also a fan of AI then?
A calculator doesn’t stroke your ego and you know that. That’s why you like and use AI.