Depends how fussy you want to get. Place each cross section of wire in the smallest hexagonal cell it can fit in. Put all the cells together in the most compact way possible like a honeycomb. That's called "Hexagonal Close Packing" and is the densest way you can fit the wires. You run into problems at the end of the layers as you indicated. One layer would have ten turns, the next nine, then ten, and so on. A machine might be able to achieve this density of packing, but it's unlikely you could do it free hand.
