How do I determine item is offline or not?

I am kind of wondering how do I know item is offline or not? will item icon shows rather than on/off?

It depends on the hardware of your item.
If your item hardware has an IP adress, you could use

http://docs.openhab.org/addons/bindings/network/readme.html

If your Item gets refreshed periodically at a set interval, you can use the Expire binding to reset the Item to NULL when it hasn’t received an update after a certain period of time.

Thanks for the answer, I need to study a while at the doc, it looks a bit complicate.