Yes, it is reappearing after deleting the thing. So atm I need to have a dummy thing with this wrong app name and the power switch linked to it. Not nice, but I have a workaround. Would be great if you can find a permanent solution.
Edit: I found this topic about removing retained messages, might be helpful…
Clearing MQTT retained messages - Tutorials & Examples - openHAB Community