I think there is an ESPHome add-on on the marketplace.It seems like adding audio support to that would be easier than creating a whole new add-on.
But in general, an add-on is probably the “correct” approach.
I think Willow - Open Source Echo/Google Home Quality Speech Hardware for $50 is based on the ESP32-S3 too so there might be something you can take from there as an alternative to ESPHome or in addition to ESPHome.