I don't have a Retina display, but I suspect your monitor is exposing a limitation in the way in which the tragic wand tool works. The algorithm used creates a black and white selection only with no greys for transitioning at the edges and its only by having the edges later ant aliased that the mask looks acceptable. This however does mean the magic wand tool is very good at detecting fine detail, if not so good at masking it. If you use the Quick Selection tool or color range which do create selections/masks with greyscale transitioning the edges are much less jaggy. If you look at the wonderful work of art below you can see this happening. The mask on the left was made with the magic wand and on the right the quick selection tool. You can see the magic wand picks up the really fine wires around the gondola and also the flag, but at the expense of the whole mask being far more contrasty.
In answer to your question, I think your just seeing the tool working in the way in which it was designed too.
