That's pretty much the gist of it, either you enjoy it, or you have other things to do.
I love it. I never thought I would be a guy with 50lbs of gunpowder and a couple thousand projectiles in his basement, but after seeing the quality of rounds I can make, I'm sold for life.
If you still have 50# of powder powder in the basement, you need to get hopping and turn that into more of them quality rounds!
"I'm going quit hand loading and find another 40 hour a week job" Said no one.