Yes, you need yo go into the /icons/classic folder and find the contact-open.png and the contact-closed.png
Copy them and name the copies contact-on.png and contact-off.png
EDIT: @Dim pointed out that the doc’s also mention this:
"Some icons are dynamically selected by openHAB depending on the Item’s state. For example, a “switch” icon may appear to be green when the Item is “ON” and red when the item is “OFF. Behind the scenes, openHAB is actually selecting two different icon files depending upon the Item state - switch-on or switch-off . A third default icon file, switch , is required as well. This icon file matches when none of the other icon files match the Item state (e.g. when the Item is in an undefined state).”
So the original naming above should work.
If your still having issues, this shouldn’t be required but, maybe try adding the icon on your sitemap opposed to the item.
As Dim mentioned, try changing the name by simply adding my to the first part of the name like mycontact-on.png for the file and use mycontact-on in either the item or sitemap.
If all your icons are .png (bitmap) type then a change to the UI config may be needed. This is done in PaperUI > Confgiuration > Services > UI > Basic UI > Icon Format. Select bitmap for .png and vector for .svg
Note: If your using both types and all are located in /etc/openhab2/icons/classic then no change is needed.