What kind of girls are you into?
Then, you can decide how big / toned you need to be. Usually girls have a preference and sometimes it is hard to tell. But you might meet two pretty much the same girls but one is short and the other is regular height. The short one might prefer you skinny and think it is perfect because you are not overboard intimidating for her, while the other with regular height will think you are alright.
And then you meet a short girl who's father has trunks instead of arms and wants an Arnold type physique guy.
Point being, if you are Brad Pitt in Fight club kind of fit, then you are probably in the sweet spot for most girls. I read somewhere that having proportions like 63cm thighs, 85cm waist, 104cm chest, 32 cm arms while at 185cm tall is considered as most attractive proportions. The numbers are a bit rough because I remember the thighs only, the rest is based on the fact that at one point I needed like 10-15% more to reach that. I am not sure where I read that either. But there is one more thing - how comfortable do you feel in your own skin? Do you need your arms to be an inch bigger or your waist smaller, do you feel you need thicker neck? Because once you feel great about yourself, girls will feel that from you simply based on the fact that you feel satisfied with yourself.
However, I agree that at the end of the day, game trumps this.