It is all about balance. If you are successful then you can end up with too much work.
What about the family? When we call a plumber we have different experiences with different plumbers. Of course we want a plumber now. The plumber that can come over now gets the work over somebody that has an answering machine.
What do you value most? Time off with just a regular job? Regular job plus p/t business and more money? If you have a family discuss this with them.....
