Tech Watch | InfoWorld Staff » MS antisphishing tool wins MS bakeoff

September 28, 2006 | Comments: (0)

MS antisphishing tool wins MS bakeoff

TAGS: Security

Microsoft sponsored a study comparing the effectiveness of antiphishing technologies and, surprise surprise, the company's IE 7 anti-phishing technology came out on top, according to a post on the IE Blog.

The study, which was conducted by 3Sharp, compared antiphishing toolbars from Microsoft, NetCraft, Google/Firefox, AOL, EarthLink, eBay, Geotrust, Netscape and McAfee. The study used a "standardized set of 100 known phishing Web site URLs and 500 known good URLs to see how well each anti-phishing technology flagged both phish and legitimate URLs."

Antiphishing toolbars were evaluated by "how well it did two things: warn or block the user from actual live phishing Web sites, and refrain from incorrect warnings or blocking on legitimate Web pages."

Toolbars were rated on a scale from 0 to 200, where "0" is the equivalent of having a browser with no antiphishing technology and 200 is a perfect antiphishing product that "caught all the known phish without making any mistakes by falsely warning or blocking any good URLs as phish."

Microsoft's Phishing Filter (MPF) in IE 7 Beta 3 received the highest "composite score" at 172, followed closely by NetCraft's toolbar with a composite score of 168.

But when you dig into the numbers, another story emerges. First of all, IE's MPF antiphishing toolbar doesn't top out any of the individual tests that make up the composite score. It finished second to GeoTrust's toolbar in spotting known phishing URLs (%89 percent catch rate, compared to %99 percent for GeoTrust). True, it didn't misidentify any known-good Web sites, but neither did five of the other toolbars tested.

So how did MPF end up on top? It boils down to how 3Sharp calculates that "composite score." As the group says, it intends the composite score to represent how good the phishing toolbar was compared with no protection at all. So the survey assigned greater points for blocking malicious sites than just warning about them, and more for warning about them than for doing nothing. On the flip side, blocking or warning on a good site cost more than doing nothing about a known good site.

So in the end, Microsoft didn't do the best job of spotting phish sites, but it did do the best job of blocking the ones it did spot, and blocking was what garnered the most points. In contrast, GeoTrust found almost all the phishing sites that were thrown at it, but doesn't have a blocking capability, and only warns users. GeoTrust also stepped in it, big time, when it came to false positives: a whopping %32, compared with just %1.6 for the next highest contender, EarthLink, and %0 for everyone else. That's outrageous!!

Moving on...NetCraft blocked all %84 percent of the sites it correctly identified, and that was better than Microsoft's %83 block rate, but Microsoft warned on another %6 percent that NetCraft didn't so...Microsoft WINS!!!!

Unfair? Possibly. Blocking a phishing Web site earned you twice as many points as just warning about it in this test, but is blocking really twice as effective as just warning users? That's reasearch that needs to be done. It certainly seems like 3Sharp's study may have been an outcome in search of a method -- but that wouldn't be anything new for vendor sponsored studies. After all, we can assume that if the numbers hadn't come out favoring IE7, we probably would never have seen this study. But hey, it's Microsoft's money.

But while we're at it, why not talk about the stinkiest antiphishing technology. According to 3Sharp, that honor belongs to McAfee's SiteAdvisor, which recorded a composite score of...hold your breath: "3." That's right: 3.

Shane Keats, of SiteAdvisor cries foul on that. "It's silly and wrong. We don't claim, anywhere, to offer phishing protection. In fact, we're pretty explicit that we don't."

True enough. SiteAdvisor is more of a malicious Web site detection service that can spot "fishy" rather than "phishing" Web sites -- developing a kind of reputation service for Web sites of all sorts -- will they misuse your personal inofrmation, have they distributed spyware or other malicious code, etc. Phishing and ID theft are part of that...kind of...but its just a piece.

Posted by Paul Roberts on September 28, 2006 09:55 AM

RATE THIS ARTICLE: