If you get a Zigbee smart switch (or smart bulb) and a Zigbee remote, you can pair the switch/bulb and remote directly so it works like you said, while still retaining the ability to control the light using Home Assistant (eg automatically turn it on or off based on something).