Even when the lights are off, the neutral wire provides power to the light switch. A neutral wire doesn't control the lights. It's possible that the neutral wire wasn't connected to your previous switch.
Let's Get It Fixed!
HomeX virtual experts will assess your issue (for free!) and can resolve simple problems.