For OH to control the lights it will need a way to communicate via zigbee. There is a zigbee binding here is the documentation. You will need to buy a hub or USB stick for the actual comm part. Also, check the list of devices that will work and make sure what you have is on the list.
I have a few Sengled lights that OH controls via zigbee2mqtt but I’ve never tried the zigbee binding. See this link and scroll down to Getting Started for what’s needed and how to. What I like about zigbee2mqtt is you can add most device that uses the zigbee protocol. In the same link it describes how to add devices that are not already listed. If you choose this route I recommend buying your USB and flash hardware from China (alliexpress) as the cost is around $25 for everything.
For Alexa see Released: Openhab2 Amazon Echo Control Binding (controlling alexa from openhab2) for how to use with OH. Its a long topic but try using the search function (Ctrl F), within the topic, to see if there is a way to control your lights via the binding and Alexa routines.
Hope this small bit of info helps and regardless what route you take there will be someone here that can help if you get stuck.