Thanks! So what I did to fix this was replace \userdata\etc\log4j2.xml with the official file: openhab-distro/distributions/openhab/src/main/resources/userdata/etc/log4j2.xml at main · openhab/openhab-distro (github.com)
My entire userdata folder is in a volume so it gets transferred from one version to the next. If Openhab doesn’t have any migration rules on that file then it won’t get updated.