Yes, I’m aware of what they tested and what fractions are…
I was just expanding the point to other things that are comparable. And an LLM for a reading comprehension test is exactly what a large language model would be best suited for, so in essence a language “calculator”.
Doesn’t change the rest of the discussion, which was about the brain’s ability to jump back into things when the crutch is taken away.
I don’t think it’s fair to say that fractions are similar to trigonometry and calculus aside from the fact that they’re all types of math
Very few people do trigonometry or calculus in their heads, I’m sure most people who know it really well could do a cosine on paper, but that’s a much higher level operation than a fraction
Um, you know all of those were developed before calculators, right?
I learned both trig and calc on the principles in how they were derived, so yes, when I was learning and using them, I could do some of it in my head or with a few figures on paper.
But I never said “similar” at all, I was using them as comparable subjects that could be done by hand or with aid.
You’re so obsessed with calling out someone on not mentioning fractions you’ve missed the point of the test, article, and this thread. They use fractions because it was above simple math but not so high level that some of the random people picked might not know it. Which going back to the real discussion, having randoms whose skill isn’t known even halfway through the test might have been a flaw in the test.
Um, you know all of those were developed before calculators, right?
What does the timing of the advent of the calculator have to do with what I said?
But I never said “similar” at all, I was using them as comparable subjects
I would contest that they are comparable subjects. Entry level piano lessons and music theory are both technically of the same subject, but wildly different in practice. One is suitable for beginners or lay people, the other is not.
You’re so obsessed with calling out someone on not mentioning fractions you’ve missed the point of the test, article, and this thread.
You are projecting your own obsession of using the poor comparison of “AI = Calculator” onto me. That’s why I keep bringing up fractions, it is an easy-to-understand (for most) rebuttal to your logically fallacious proposal.
They use fractions because it was above simple math but not so high level that some of the random people picked might not know it.
Exactly, that’s why your comparison doesn’t work.
Which going back to the real discussion, having randoms whose skill isn’t known even halfway through the test might have been a flaw in the test.
What changes would you make to the experiment to remove the flaw you’re perceiving?
How do you expect that change would impact the outcome?
As I’ve said before, the abilities of the ones with ChatGPT assist don’t seem to be known, as they started off with help. The assumption is that they might on average be the same as the others, but we don’t know that. A simple fix to start is to offer the LLM help in the middle of the test for them, see how the curve of ability goes up and down, and compare before and after.
Not answering the rest, I was more interested in the study and what it was claiming and not arguing about missing context of commentary. I think the study may be right in suggesting people today have more trouble understanding things than they used to, but it’s far deeper and older than LLMs. They are just the latest part of the bigger problem.
I don’t think that change would negatively impact the experiment, but for what they were trying to measure I’m unsure of how necessary that step would be.
They weren’t measuring how smart or good at math the participants were, they were specifically measuring changes in focus and persistence (so basically the amount of effort put forward). The findings show that people tend to just give up when you take the LLM away, and I think that with your proposed change you’d still get largely the same findings.
I would agree that LLMs aren’t the entire issue, but they are making the issue far worse at an exponential rate.
I’d be interested to read a study about what factors and/or traits predispose someone to trust an LLM, or allow themselves to become dependent on it.
They weren’t measuring the math ability levels, but the changes in what those skills were per individual. But for the assisted ones, we don’t know what they started with, so there’s no base level to compare. If they’re comparing them with the others, then that’s a problem in the study since we’re not measuring their math abilities.
What would a companion study reveal that does the same thing for the math part but just offers calculators instead? Probably the same? If it was dramatically different then there’d be a case that LLMs are affecting the result in some other way and it’s not just the human nature of leaning on what they lost. Or what if that person had a tutor/help with them but they left for the rest of the test? Should be similar results, right?
What would a companion study reveal that does the same thing for the math part but just offers calculators instead?
I think that depends on the difficulty of the math in question. If it was something that most people don’t know how to do by hand (like your trigonometry example) you’d probably see a more severe drop off.
I believe they chose fractions because it’s something that everyone who’s gone through the public education system should know, and they’re more relevant to (most people’s) lives than trigonometry (people use them to split bills, calculate tip, etc.)
If you got a group of people who all should know trigonometry (due to their profession perhaps), and then ran the calculator test, it would be more analogous.
What would a companion study reveal that does the same thing for the math part but just offers calculators instead? Probably the same?
That would not be my hypothesis personally, but perhaps I’m biased, I intentionally do most of my basic math mentally and only use the calculator for more complicated equations. If I was in this study I would want to see how much of it I could do without the calculator first… perhaps for the study the calculator use would be compulsory, but I wouldn’t just give up when it was taken away if I didn’t need it in the first place.
Or what if that person had a tutor/help with them but they left for the rest of the test? Should be similar results, right?
I think that would depend on the knowledge the subject had before entering the testing scenario, and also on the quality of the tutor in question. Personally, I don’t always approach things in the conventional way, so a tutor could be more of a hindrance than a help if they disliked my methodology.
I think that would depend on the knowledge the subject had before entering the testing scenario
That was my main point on this test being an interesting question starter, but not being conclusive enough because we don’t know. The subjects were purposefully random and not evaluated before the test.
fractions because it’s something that everyone who’s gone through the public education system should know
Yes, well. I don’t think I need to comment on how bad the US system has become. I wonder how much higher the assisted people did while they had the help compared to others?
Yes, I’m aware of what they tested and what fractions are…
I was just expanding the point to other things that are comparable. And an LLM for a reading comprehension test is exactly what a large language model would be best suited for, so in essence a language “calculator”.
Doesn’t change the rest of the discussion, which was about the brain’s ability to jump back into things when the crutch is taken away.
I don’t think it’s fair to say that fractions are similar to trigonometry and calculus aside from the fact that they’re all types of math
Very few people do trigonometry or calculus in their heads, I’m sure most people who know it really well could do a cosine on paper, but that’s a much higher level operation than a fraction
Um, you know all of those were developed before calculators, right?
I learned both trig and calc on the principles in how they were derived, so yes, when I was learning and using them, I could do some of it in my head or with a few figures on paper.
But I never said “similar” at all, I was using them as comparable subjects that could be done by hand or with aid.
You’re so obsessed with calling out someone on not mentioning fractions you’ve missed the point of the test, article, and this thread. They use fractions because it was above simple math but not so high level that some of the random people picked might not know it. Which going back to the real discussion, having randoms whose skill isn’t known even halfway through the test might have been a flaw in the test.
What does the timing of the advent of the calculator have to do with what I said?
I would contest that they are comparable subjects. Entry level piano lessons and music theory are both technically of the same subject, but wildly different in practice. One is suitable for beginners or lay people, the other is not.
You are projecting your own obsession of using the poor comparison of “AI = Calculator” onto me. That’s why I keep bringing up fractions, it is an easy-to-understand (for most) rebuttal to your logically fallacious proposal.
Exactly, that’s why your comparison doesn’t work.
What changes would you make to the experiment to remove the flaw you’re perceiving?
How do you expect that change would impact the outcome?
As I’ve said before, the abilities of the ones with ChatGPT assist don’t seem to be known, as they started off with help. The assumption is that they might on average be the same as the others, but we don’t know that. A simple fix to start is to offer the LLM help in the middle of the test for them, see how the curve of ability goes up and down, and compare before and after.
Not answering the rest, I was more interested in the study and what it was claiming and not arguing about missing context of commentary. I think the study may be right in suggesting people today have more trouble understanding things than they used to, but it’s far deeper and older than LLMs. They are just the latest part of the bigger problem.
I don’t think that change would negatively impact the experiment, but for what they were trying to measure I’m unsure of how necessary that step would be.
They weren’t measuring how smart or good at math the participants were, they were specifically measuring changes in focus and persistence (so basically the amount of effort put forward). The findings show that people tend to just give up when you take the LLM away, and I think that with your proposed change you’d still get largely the same findings.
I would agree that LLMs aren’t the entire issue, but they are making the issue far worse at an exponential rate.
I’d be interested to read a study about what factors and/or traits predispose someone to trust an LLM, or allow themselves to become dependent on it.
They weren’t measuring the math ability levels, but the changes in what those skills were per individual. But for the assisted ones, we don’t know what they started with, so there’s no base level to compare. If they’re comparing them with the others, then that’s a problem in the study since we’re not measuring their math abilities.
What would a companion study reveal that does the same thing for the math part but just offers calculators instead? Probably the same? If it was dramatically different then there’d be a case that LLMs are affecting the result in some other way and it’s not just the human nature of leaning on what they lost. Or what if that person had a tutor/help with them but they left for the rest of the test? Should be similar results, right?
I think that depends on the difficulty of the math in question. If it was something that most people don’t know how to do by hand (like your trigonometry example) you’d probably see a more severe drop off.
I believe they chose fractions because it’s something that everyone who’s gone through the public education system should know, and they’re more relevant to (most people’s) lives than trigonometry (people use them to split bills, calculate tip, etc.)
If you got a group of people who all should know trigonometry (due to their profession perhaps), and then ran the calculator test, it would be more analogous.
That would not be my hypothesis personally, but perhaps I’m biased, I intentionally do most of my basic math mentally and only use the calculator for more complicated equations. If I was in this study I would want to see how much of it I could do without the calculator first… perhaps for the study the calculator use would be compulsory, but I wouldn’t just give up when it was taken away if I didn’t need it in the first place.
I think that would depend on the knowledge the subject had before entering the testing scenario, and also on the quality of the tutor in question. Personally, I don’t always approach things in the conventional way, so a tutor could be more of a hindrance than a help if they disliked my methodology.
That was my main point on this test being an interesting question starter, but not being conclusive enough because we don’t know. The subjects were purposefully random and not evaluated before the test.
Yes, well. I don’t think I need to comment on how bad the US system has become. I wonder how much higher the assisted people did while they had the help compared to others?