The way to find the best line of fit would be to place a straight line through the middle of the majority of the points, wherever they may be. Outliers throw this off and sometimes the outliers won't be included.
Do you have an example you'd like help with?
