The issue is that it isn’t a straight shot as a lot of people paint. Call Centers work off of User Interfaces, AI can’t see or use those, so those UIs suddenly have to be retooled in a way that the AI understands, which that’s not easy. Additionally there’s business logic that is complex and there’s a lot of siloed knowledge, all of that is hard to extract and put into a model that’s usable.
The thing is that these LLM and AI companies were thinking the rest of the world is as structured as the data models they trained their AIs on and that’s just not the case. The LLMs can absolutely do the task if given the task correctly, it just that it’s near impossible to give the task they need to perform correctly in 100% of the situations. Hell, even humans fail this, people get written up at call centers all the time.
To put it simple, you ever hear the joke, “we don’t have to worry about AI taking the programmers jobs because then the CEO would have to accurately explain the problem they’re trying to solve/sell”? It’s IRL that, that’s holding up a ton of the LLMs in call centers. Like there’s two VERY narrow processes that the company I work for has implemented AI for, and those are really basic situations where explaining the full scope is pretty easy.
But take what I have to say with a grain of salt. I can’t say the company I work for has ever really been that gung-ho about AI to begin with. But I can tell you that it’s WAY, WAY, WAY more work to deploy AI than the tech bros like to paint it. Like you can just hit the button and “go”, but it’s going to crash and burn. Like to get it right is way more work than the AI industry let’s on.
The thing about this though, is that it’s not a new problem at all. LLMs didn’t start to get good enough in the early 20’s and only then did they come up with this idea. I worked for a company out near Seattle back in ~2014 that was already well into trying to tackle this problem.
They ran callcenters with a variety of contracts for different companies and took calls, chats, and emails. The main business model wasn’t the centers themselves but the information gathered by the ticketing system to help build tools like this.
Personally, with that insight and assuming surely there must’ve been other companies moving along that path, I find it quite telling that they still haven’t sufficiently stepped up to the role. There are some hard limits on cost and hallucinations that I think will ultimately fail to deliver a truly long-term, viable product. When you see they can’t maintain the veneer on even that use case, you’ll know the bubble has to be close to popping.
Of course no one can really say for sure, we’ve all been predicting it for some time and when there’s this much money invested they’ll protect that reality ferociously, so who really fucking knows. But still …
What you’ve said makes me think that LLMs have a great use case in creating and searching documentation but if anyone is calling, it really needs a person to deal with that edge case
Not creating and searching as far as I understand (or as far as I’m willing to allow it to in this case) but more summarizing, truncating, some times of rewording (non-technical parts).
They’re getting better at extracting information out of a closed set of data, but it’s still literally impossible to guarantee that it won’t generate a contradictory or unwanted piece of text that looks very close to the right thing, based off the training data inherent to the model.
But the “best” case is something closed ended where you know what the output is. So cleaning up a tiny piece of code, summarizing something that you provide in its entirety, translating a block of text, that’s all a good use case. Using it to distill the entire web’s information into a chatbot format? Fuck no
The entire problem is people thinking this tool that can turn text input into soup and reliably pull text back out of said soup is something it just is not.
Most of the models I’ve played with before the boom were not instruct models. So you didn’t prompt them and have them churn out slop that sounds like the answer to your question. Instead you just wrote text (story, article heading, etc) and it would continue the pattern. The results were “worse” in quality, but because we only thought to use it a specific way, it felt like a very powerful new tool.
My enthusiasm for this shit has fallen through the floor in 2022 and presently is about 18% through the earth’s outer crust
You can have LLMs draft documentation based on callcentre communication - that’s what I meant by creating
With regards to searching, the thinking models seem really good at finding what you need when you don’t know what exactly to search for
“Semantic search” / “semantic indexing”. Yes. Would be a great thing to optionally have. But you don’t need to hook it up to a prompt and have it spit out natural language output.
It could be just like a standard search with search results, just with a backend that looks at more stuff based on meaning not just explicit word matching. And search engines have worked like this for years to be fair.
But I agree, the general purpose chatbots are probably helpful to get a foothold on looking something up when you don’t really know what it’s called or how to concisely describe it. The problem is that the companies that make them have every incentive to feed you their explanation too, not just point you in the right direction and have you leave their service.
Ish.
The issue is that it isn’t a straight shot as a lot of people paint. Call Centers work off of User Interfaces, AI can’t see or use those, so those UIs suddenly have to be retooled in a way that the AI understands, which that’s not easy. Additionally there’s business logic that is complex and there’s a lot of siloed knowledge, all of that is hard to extract and put into a model that’s usable.
The thing is that these LLM and AI companies were thinking the rest of the world is as structured as the data models they trained their AIs on and that’s just not the case. The LLMs can absolutely do the task if given the task correctly, it just that it’s near impossible to give the task they need to perform correctly in 100% of the situations. Hell, even humans fail this, people get written up at call centers all the time.
To put it simple, you ever hear the joke, “we don’t have to worry about AI taking the programmers jobs because then the CEO would have to accurately explain the problem they’re trying to solve/sell”? It’s IRL that, that’s holding up a ton of the LLMs in call centers. Like there’s two VERY narrow processes that the company I work for has implemented AI for, and those are really basic situations where explaining the full scope is pretty easy.
But take what I have to say with a grain of salt. I can’t say the company I work for has ever really been that gung-ho about AI to begin with. But I can tell you that it’s WAY, WAY, WAY more work to deploy AI than the tech bros like to paint it. Like you can just hit the button and “go”, but it’s going to crash and burn. Like to get it right is way more work than the AI industry let’s on.
The thing about this though, is that it’s not a new problem at all. LLMs didn’t start to get good enough in the early 20’s and only then did they come up with this idea. I worked for a company out near Seattle back in ~2014 that was already well into trying to tackle this problem.
They ran callcenters with a variety of contracts for different companies and took calls, chats, and emails. The main business model wasn’t the centers themselves but the information gathered by the ticketing system to help build tools like this.
Personally, with that insight and assuming surely there must’ve been other companies moving along that path, I find it quite telling that they still haven’t sufficiently stepped up to the role. There are some hard limits on cost and hallucinations that I think will ultimately fail to deliver a truly long-term, viable product. When you see they can’t maintain the veneer on even that use case, you’ll know the bubble has to be close to popping.
Of course no one can really say for sure, we’ve all been predicting it for some time and when there’s this much money invested they’ll protect that reality ferociously, so who really fucking knows. But still …
What you’ve said makes me think that LLMs have a great use case in creating and searching documentation but if anyone is calling, it really needs a person to deal with that edge case
Not creating and searching as far as I understand (or as far as I’m willing to allow it to in this case) but more summarizing, truncating, some times of rewording (non-technical parts).
They’re getting better at extracting information out of a closed set of data, but it’s still literally impossible to guarantee that it won’t generate a contradictory or unwanted piece of text that looks very close to the right thing, based off the training data inherent to the model.
But the “best” case is something closed ended where you know what the output is. So cleaning up a tiny piece of code, summarizing something that you provide in its entirety, translating a block of text, that’s all a good use case. Using it to distill the entire web’s information into a chatbot format? Fuck no
The entire problem is people thinking this tool that can turn text input into soup and reliably pull text back out of said soup is something it just is not.
Most of the models I’ve played with before the boom were not instruct models. So you didn’t prompt them and have them churn out slop that sounds like the answer to your question. Instead you just wrote text (story, article heading, etc) and it would continue the pattern. The results were “worse” in quality, but because we only thought to use it a specific way, it felt like a very powerful new tool.
My enthusiasm for this shit has fallen through the floor in 2022 and presently is about 18% through the earth’s outer crust
You can have LLMs draft documentation based on callcentre communication - that’s what I meant by creating With regards to searching, the thinking models seem really good at finding what you need when you don’t know what exactly to search for
“Semantic search” / “semantic indexing”. Yes. Would be a great thing to optionally have. But you don’t need to hook it up to a prompt and have it spit out natural language output.
It could be just like a standard search with search results, just with a backend that looks at more stuff based on meaning not just explicit word matching. And search engines have worked like this for years to be fair.
But I agree, the general purpose chatbots are probably helpful to get a foothold on looking something up when you don’t really know what it’s called or how to concisely describe it. The problem is that the companies that make them have every incentive to feed you their explanation too, not just point you in the right direction and have you leave their service.