ESPHome add-on on the marketplace
That was my intention 
I think Willow - Open Source Echo/Google Home Quality Speech Hardware for $50 is based on the ESP32-S3 too so there might be something you can take from there as an alternative to ESPHome or in addition to ESPHome.
Not jumped into code, but ESPHome is prepared for customExtensions. So I hope we stay with that “Jack of all trades” 