I think a core part of it is going to be assigning the right queries to the right models.

Dont send recipe or ‘Google this for me’ queries to the latest most power hungry model, for example. Older models are already hundreds of times less computationally expensive than they were two or three years ago so the problem is already half solved in some sense.