Unfortunately, that's not the worst of the invisible man's troubles. As soon as he drank that invisibility potion, he would also fall apart.
To understand why we have to think for a moment about what light is. "Light" is just the term we use for electromagnetic energy at frequencies that we can see. Whenever we see something, our eyes are responding to the electromagnetic energy that is reaching them from some object.
But that's not all electromagnetic energy does. It's also responsible for the "opposites attract" rule. At the level of the atom, electromagnetic energy is what holds positively-charged protons to negatively-charged electrons. It also pulls together positive atoms and negative ones. These kinds of interactions are essential to keeping matter in a stable form.
Now, if the light passes right through the invisible man, that's the same thing as saying his body no longer interacts with electromagnetic energy. If that's true, the atoms that make up his body are in big trouble -- all rules are off. In an instant they would fly apart, and our invisible friend would disappear, this time for good.