Have I scrubbed the waterproof varnish off my wood?

I have cleaned my knives for years with one of those green kitchen scrubbies. They get the gunk off the blade, but they also rub off the fine steel polish, which I don't mind because I don't like my knives too shiny. But I used a scrubby on the wood handle, and now the varnish gleam is gone. Have Iscrubbed off the waterproof varnish, or just made it not shiny? I don't care if it's shiny, I just want the handle waterproof.