I am not a machinist but your plan makes sense to me.
However, going with my philosophy that every job I do needs to require me to buy a tool I do not currently own, I would suggest you ditch the drill press and get a proper milling machine.
To my way of thinking, if you are willing to these types of mods, you may find it handy tool for other jobs that come along and you can always use it to just drill holes.
(say 'YES" to targas!, please people, let's just admit they were a great idea and admire those who stepped up!
)