Why least-squares makes sense

An offhand comment in my machine learning textbook finally gave me a good intuitive understanding for why you want to square the error terms when doing a best fit of a line to a set of data. Previously, the best explanation I'd ever heard was something like "well. by squaring you guarantee that the total error is non-negative", which wasn't sufficient to explain why x2 is better than, say, x4 or abs(x).

The reason that makes sense to me is this: by making your total error be the sum of squares, the derivative of your error becomes linear. Since the derivative is linear it has a single root, which corresponds to the global minimum of the error function. So you use a sum of squares because it gives you a well-defined global minimum.

It's pretty obvious once you see this, but nobody had ever really explained it to me before.

Posted on September 18, 2006 11:14 PM
More school articles

Comments

Great explanation, thanks. Oddly enough, I had never heard a good enough explanation for this either.

Posted by: Hans at October 1, 2006 06:35 PM

Every convex functional has a global optimum, so I wouldn't say that's its raison d'etre although it's certainly part of its appeal!

Here's another bit of intuition you might like. The general setup is that we have a set of n points p_1, p_2, ..., p_n in R^k, and we are looking for the best-fit hyperplane. Take the simplest case of k=1. Then a hyperplane is simply a point q. The least-squares error is

E(q) = sum(i) (p_i - q)^2

We can find an optimum by differentiating wrt q and then solving for zero:

E'(q) = sum(i) -2*(p_i - q)

E'(q) = 0
sum(i) -2*(p_i - q) = 0
sum(i) (p_i - q) = 0
sum(i) p_i = n*q
q = (sum(i) p_i)/n

So the least-squares optimizer is just the mean of the points. This shows that the least-squares optimizer generalizes the mean to higher-dimensional hyperplanes.

Posted by: Per Vognsen at October 15, 2006 10:19 PM

Also in engineeing problems least squares method minimize the error in sense of the POWER!! E.g. in the case of an audio project, the power is what you hear, and therefore the true error.

Posted by: ulrik at October 23, 2006 07:52 AM

Here's my 2 cents...

-- From a theoretical standpoint, least squares (LS) is justified because it gives you the maximum likelihood estimate under the assumption that your measurements are corrupted by Gaussian i.i.d. noise. This is nice because in many cases the Gaussian noise assumption is acceptable

-- From a practical standpoint, LS is used because it is easy: if your model is linear in the unknown parameters, say f(x) = Ax (where A is a known matrix and x is the parameter vector you're interested in), and you pose the problem as minimization of the sum of squares of the error vector (y - Ax) , where y are your noise-corrupted measurements, the (minimum norm) LS solution is given in closed form by x = pinv(A)y where pinv denotes pseudoinverse, in other words, the solution is a known linear function of the measurements!

Posted by: diego at January 11, 2007 12:08 PM

Statisticians seem almost embarrassed to admit that, among all the reasons for using least-squares, the most important one is the simplification of the calculus behind the fitting process (which you note).

For my money, though, least absolute errors regression makes more sense, and, today, cheap and plentiful computing power makes this option nearly as easy as least-squares.

Posted by: Will Dwinnell at October 26, 2007 01:41 AM

As humans, we're simply born into one of these dog training classes: chief or follower. As these couple types are extremes of one another, you may simply be a chief, or a buff, tiny is there of any fashion in between these puppy training.Some might contest though, that you could be both at one time, the chief to numerous and a buff to one.There are more teach dog tricks to outlook out for to review if or not you're a chief or a follower. The bulk all over individual aspect is the abilities to generate determinations not simply for yourself, but in supplement for remnant as prolonged as dog behavior training.Take this covering for example; each post-graduate learner is rewarded a cubicle each, which doesn't draw seal for loose though. Most would just announce no just because it doesn't draw seal free to wholesale dog supplies. The actual justification behind their tenacity is the portion that none wants to act on away from the crowd.When one or couple of them creates a tenacity to receive one cubicle in any dog supplies, remnant in the past prolonged followed not far behind. One later another, the cubicles are being filled. Sadly, there just isn't passable room for everyone to wholesale pet supplies. Even as learners with very competent standings and privileges, not all are competent to take find how to lose weight and generate stern life correcting determinations on their own. A buff would perpetually have determinations organised for them, perpetually staring to lose weight to play-act first after the time arises. If you're one of those any person who essentially have a darling to move out with the accumulation, it's come seal time to commence correcting and take find fault for once. belly fat is a yardstick journalist for a many cluster of themes and subjects. He in supplement drafts on braun bathe and transform and braun bathe and charge.

Posted by: dogtraining at October 21, 2009 08:42 PM

Great post mate, keep it up.

Posted by: free online games at October 25, 2009 04:09 PM

These Women's Classsic Tall 5815 Boots in pink flower print can either be worn at their normal height if you want Ugg boots, but in order for you to create a more casual look to what you are wearing they can be pushed down Women's UGG Classic Tall 5815 Boots in Baked Clay

Posted by: ugg boots on sale at October 27, 2009 02:34 PM

Love this man. Great stuff, great prices, free shipping. Email him if you cant find what you are looking for and he will look for you. Good customer service.louis vuittonSomeone who is knowledgeable in louis vuitton.Another guy from hong Kong (Largest louis vuitton second hand city, I believe) he has great stuff at fair prices but his shipping is insane

Posted by: louis vuitton at October 29, 2009 08:11 AM

Love this man. Great stuff, great prices, free shipping. Email him if you cant find what you are looking for and he will look for you. Good customer service.louis vuittonSomeone who is knowledgeable in louis vuitton.Another guy from hong Kong (Largest louis vuitton second hand city, I believe) he has great stuff at fair prices but his shipping is insane

Posted by: louis vuitton at October 29, 2009 08:11 AM

Love this man. Great stuff, great prices, free shipping. Email him if you cant find what you are looking for and he will look for you. Good customer service.louis vuittonSomeone who is knowledgeable in louis vuitton.Another guy from hong Kong (Largest louis vuitton second hand city, I believe) he has great stuff at fair prices but his shipping is insane

Posted by: louis vuitton at October 29, 2009 08:11 AM

Love this man. Great stuff, great prices, free shipping. Email him if you cant find what you are looking for and he will look for you. Good customer service.louis vuittonSomeone who is knowledgeable in louis vuitton.Another guy from hong Kong (Largest louis vuitton second hand city, I believe) he has great stuff at fair prices but his shipping is insane

Posted by: louis vuitton at October 29, 2009 08:11 AM

Good day! Thx for your great post and Im thinking about how to introduce my ugg boots to you cos Im not sure if u like this. Many people who live in the cold area like the ugg boots, especially the north-europe. Our uggshare.co.uk supply the cheapest classic cardy ugg boots, class tall ugg boots, classic short ugg boots and other series of uggs, hope my words didn’t trouble u, cy!

Posted by: ugg boots at November 5, 2009 03:42 AM

Eliminating the need for the extra trouble to wear ugg boots uk
ugg boots saleuggs, in the extreme sports, look for happiness in life, with UGG Classic boots and nike sb showed off your tall body, do not worry about being laughed at, dancing new style ugg australia and nike sb dunk . The flavor of life that is colorful, there are Christian Louboutincheap nike dunksUGG cardy boots、and nike sb for sale also used fear to creep away money from your pocket Mody ? Love beauty, you might even love UGG Classic tall bootsUGG Nightfall bootsUGG Classic short bootsnew nike sb。There are an infinite high power drivers of popular front ugg boots saleugg saleuggsnike dunkdiscount ugg bootsnike dunk sb, our website also have UGG Nightfall bootsbutton ugg bootsbailey button uggugg button bootsbailey button bootsnike dunk midnike dunk highChristian Louboutin Boots, pick your favorite, do not hesitate!

Posted by: ugg boots sale at November 12, 2009 03:44 AM

Uggs on sale now.UGG Classic Cardy Boot makes me different form the other

girls. The UGG Bailey Button Boot is a good choice for female.

Posted by: ugg classic cardy boots at November 17, 2009 12:02 AM

Louis Vuitton , commonly referred to as Louis Vuitton 08 FW Senior RTW Show and Louis Vuitton Limited Sports Car, or sometimes shortened to Louis Vuitton & Infiniti In 2009 Geneva Motor Show has become one of the most Luis Vuitton bags Agendas luxurybrands Louis Vuitton Change Flower Into ATM.

Posted by: 1 at November 17, 2009 01:19 AM

UGG Bailey Button bootsis a new style in 2009.The classic cardy uggs boots is another hot boots that worth of buying.And the classic tall ugg boots will make your winter amusing.And now uggs on sale,if you are looking for such a boot,the ugg boots is good choice this year.

Posted by: UGG Bailey Button Boots at November 17, 2009 01:46 AM

I love Women's UGG Classic Cardy Boot 5819 very much. And the new style UGG Mayfaire Boots is aslo a good choice this year.

Posted by: UGG Bailey Button Fancy Boots at November 17, 2009 03:20 AM

lxjuan08ed hardyis a famous ed hardy store which sell directly ed hardy clothing, shoes, boots, swim suit and other cheap ed hardy.


Posted by: ed hardy at November 18, 2009 03:31 AM

lxjuan08ed hardyis a famous ed hardy store which sell directly ed hardy clothing, shoes, boots, swim suit and other cheap ed hardy.


Posted by: ed hardy at November 18, 2009 03:32 AM

lxjuan08ed hardyis a famous ed hardy store which sell directly ed hardy clothing, shoes, boots, swim suit and other cheap ed hardy.


Posted by: ed hardy at November 18, 2009 03:34 AM

We are the best online sales for the china wholesale . Here you can have a large of choices of kinds Ugg Boots,Converse Shoes,Timberland Boots,puma shoes,Nike Shox Shoes ,Nike Dunk SB Shoes,Nike Air Max,Links Of London,Tiffany Jewelry,Dior Handbags?,jimmy choo handbags ,Cartier Watches, 8GB Mp4 Players,Bluetooth Car DVDs. All our cheap online cheap goods are high quality and original packages, and best service. We offer our customers the best service, 7 days arrive at your door.Enjoy your easy and happy shopping with us.

Posted by: cheap goods sale. at November 20, 2009 12:24 AM

We provide a variety of Wedding dress gowns according to the measurements your supplied.

A view of the world's longest wedding dresses train made by Zhao Peng to please his ·

Dennis wholesale wedding dress factory would like to thank you for visiting our site and welcome wholesale wedding gowns from us.

Posted by: cokefish at November 20, 2009 02:36 AM

People all over the world know the abercrombie and fitch,but not everyone really knows how fashion the abercrombie is,hollister is the Legend maker. Everybody wears the hollister clothing would be the abercrombie mens and the abercrombie womens, if you want know you can search the Ruehl No.925 or abercrombie outlet in the www.google.com .

Posted by: fitch at November 21, 2009 03:49 AM

Your article is write very well, I like it very much ~
ghd Hair Straightener
ghd Hair Straighteners
ghd straighteners
GHD
hair straighteners
bose in ear headphones
bose on ear headphones
bose headphones
nike sb dunks
nike dunks
I wish you have a wonderful day!Thank you.

Posted by: ghd straighteners at November 22, 2009 08:44 PM

Timeberland boots on sale now! Do you want moreTimeberland boots information?Just move your mouse and press it to our Timberland shoes online store.

Posted by: Timberland uk at November 23, 2009 01:25 AM

Laptop Battery Laptop Battery Laptop Batteries
Laptop Batteries discount laptop battery
discount laptop battery
notebook battery notebook battery
computer battery computer battery
replacement laptop battery replacement laptop battery
notebook batteries notebook batteries

Posted by: Laptop Battery at November 24, 2009 10:15 PM

chaoying nfl jerseys
Giants Jerseys
Patriots Jerseys

Posted by: jerseysleague at November 29, 2009 04:12 AM

Paul Smith has been collaborating with a few different organizations recently and his latest venture sees him teaming up with Japanese bicycle saddle, Paul Smith bag, on a limited edition leather saddle. The release features Smith’s signature stripes where only 20 will be available at Paul Smith Clothing store in Japan and 8 at the Floral Street store in London. Get on it quick

Posted by: fdfe at November 29, 2009 11:12 AM

Gucci is famous as top quality, Louis Vuitton Tivoli PM luxurious, modern, and sexy luxury goods. Louis Vuitton Replica Tivoli PM It is a symbol of status and wealth. From time to time, Gucci belt is favored by most of people very much.As we know, Gucci belts stand for fashion trend,
Louis Vuitton Tivoli PM Replica popular elements, culture, conciseness gucci and noble. As a gentle and fashion business man or woman, Gucci belt is your best choice. Gucci commercial belts adopt real leathers, and the leathers have high hardness, toughness and scratch resistant. Belt buckles adopt high quality CTB alloy, the golden and silver color of them will never be faded. You like casual clothes and belts, because casual living Speedy 25 lifestyles focus on the simple pleasures of life. We believe that you will not hesitate to choose Gucci belt. Gucci casual belts own fashion and lovely style, bright color. The material of belt is high quality real leather or durable PU leather. Therefore, if you own this kind of Gucci casual belt, you will become a popular star.

Posted by: fanqin at December 1, 2009 12:54 AM

ugg boots sale at a time when large heat has set off a winter 08/09 to Ewha.Outlines the

warm wool winter fashion degree to wool crocheted wool ugg boots uk to join after the

body does not stimulate your skin soft, soft comfort UP. Fashion can not without
ugg boots presence in Winter.and ugg boots have many series.such as:
ugg classic
UGG Bailey Button boots
ugg nightfall
ugg kids

Posted by: uggboots at December 2, 2009 07:50 PM
Post a comment









Remember info?




Prove you're human. Type "human":