Drilling a hole where you want is a basic skill in any workshop and the standard practice is to use a center punch or awl to give the drill bit a place to start.
Layout your hole with a marker of some sort, use an awl or other sharp object to create a starting point for the drill and then drill. You might try drilling your hole early and carve the design true to the hole. Or pick/create a design that doesn't need a hole.
Drills like to pull away from center in many materials. A drill press and some sort of solid tie for the object are the best bet in that case.
Try a different drill bit. Drill bit quality makes a big difference. Hardware store drill bits have been real hit and miss in my experience. Machine tool suppliers have much better drill bits.
Drill a small hole and enlarge to final size.
If it makes you feel better I've drilled thousands of holes and usually do just fine but the other day I made one off center in a pendant blank.