Jaa

Anti-Phishing Accuracy Study

  • Artikkeli

As we’ve worked on the new Phishing Filter in IE7, we knew the key measure would be how effective it is in protecting customers. In addition to our internal tests, we wanted to find some external measure of our progress to date as well as pointing to ways we could improve. We didn’t know of a publicly available study covering the area, only some internal and media product reviews. (We’ve blogged a few times about the new Phishing Filter in IE7; in addition to these technical details we published the results of a 3rd party privacy audit.)

To help us answer this question, we asked 3 Sharp LLC to conduct a study of the Phishing Filter in IE7 along with seven other products designed to protect against phishing threats. In order to establish an accurate methodology on a level field, they utilized four sources of independent data that are not used to populate the IE7 Phishing Filter service today. They worked hard to build large enough sample sizes of actual phishing sites to draw meaningful conclusions.

3Sharp LLC tested eight browser-based products to evaluate their overall accuracy in catching 100 live confirmed phishing websites over a six week period (May – July 2006) and also understand the false-positive error rate on 500 good sites. In addition to IE7, the toolbar and browser solutions tested included the offerings from EarthLink, eBay, GeoTrust, Google Safe Browsing using Firefox, McAfee SiteAdvisor, Netcraft, and Netscape. You can see actual version numbers in the detailed report.

We are pleased to see that Internet Explorer 7’s Phishing Filter finished at the top of 3Sharp’s list as most accurate anti-phishing technology, catching nearly 9 out of 10 phishing sites while generating no warning or block errors on the 500 legitimate websites tested. You can read the report for yourself at 3Sharp’s website. The report contains details on the methodology, the data sources used and even a list of every single URL tested.

It’s great to see so many companies looking for different ways to address the significant problem of phishing. We think that the results reported by 3Sharp validate the unique approach we’ve taken of combining a service-backed block list with client-side heuristics. That said, we understand that the threat posed by phishing is constantly evolving as are the tools designed to protect users, so this set of results represents only the relative performance during that period. We know we need to keep working to keep up with the changes in the attacks and are already using the results of this test to further improve the efficacy of the Phishing Filter.

If you’re using IE7 but not already using the Phishing Filter, I encourage you to turn it on (you can find it under the Tools icon) and browse with more confidence. If you’re not using IE7 yet, you can install our latest version here.

Tony Chor
Group Program Manager

Comments

  • Anonymous
    January 01, 2003
    Nice :D

  • Anonymous
    January 01, 2003
    Keep up the good work team.

  • Anonymous
    September 28, 2006
    With Microsoft AntiPhish nothing can fool me. Thank you for AntiPhish!! I feel more confident clicking random things now without care. Microsoft, you are GREAT. Innovation! Best CSS support. Best record of blocking spyware. Good GUI. Thank you.

  • Anonymous
    September 28, 2006
    Really funny is the score of McAfee Site Advisor scoring 3 points out of 200 where the top two products including IE7 score 172 and 168.
    Hilarious even !!!

  • Anonymous
    September 28, 2006
    Congratulations, guys. Those are some pretty impressive scores. Poor McAffee. :)

  • Anonymous
    September 28, 2006
    3sharp, a Redmond based technical services company, has been commissioned by Microsoft to undertake a

  • Anonymous
    September 28, 2006
    Tony,

    I will definately turn it on for my parents and non-technical friends.  However I am not convinced it wont slow down my browsing.

    Could you give us an outline in technical detail of what steps are taken to ensure this does not slow down browsing.

  • Anonymous
    September 28, 2006
    Good work! However, I must note that my installation of RC1 identified a Microsoft web site under microsoft.com as a phishing site. So either it's a false positive or... ? ;-)

    (Unfortunately, I no longer have the link.)

  • Anonymous
    September 28, 2006
    The comment has been removed

  • Anonymous
    September 28, 2006
    PingBack from http://mozillalinks.org/wp/2006/09/microsoft-on-anti-phishing-we-are-winning/

  • Anonymous
    September 28, 2006
    Everyone: thanks for the comments far!

    David: In order to minimize the performance impact, we do the phishing filter checks asynchronously from the navigations, that is we don't block navigation while we check. If you prefer, you can also run the Phishing Filter manually, by turning PF off and then choosing "Check this website" from the Phishing Filter option under the Tools menu. This allows spot checking of sites you think might be suspicious.

    Francis: We're continuing to tune the Phishing Filter heuristics to minimize false positives to prevent good sites from being flagged.

    Sheep and Duck: I'm not sure how to convince you to trust the study; it doesn't help us improve the product if we weight the results. Ultimately, you should be the judge based on your personal experience with the tools.

  • Anonymous
    September 28, 2006
    Sheep and Duck, I understand why you're skeptical. No matter who commissioned the study, someone would distrust the results on that basis alone. However, I think if you read the report, you'll see that we have been transparent about our test methods and the data we used for the test. If you read the report and still have questions, feel free to contact me via e-mail (paulr@3sharp.com) or my blog (www.robichaux.net/blog) and I'll do my best to address them.  

  • Anonymous
    September 28, 2006
    I would be very interested to know, in cold hard facts (not a comissioned study), how many end users have been "scammed" by a phishing site? per month, per year, and ideally by geographic location.

    As an avid user of the Internet for many years, it would take a heck-uv-a-lot of a scam, to actually get me to part way with my money, or be convinced that the site was legitimate.

    7 times out of 10, the url gives it away, or the graphics, or a complete mis-guided attempt at English. ie. "For our security, your credentials you must enter below and click submit"

    Profesional web sites, just don't have glaring gramatical errors on them (or if they do, they are cleared up within minutes of posting)


    My other comment, is a request.  Can the phishing filter be added as a right click option on the url?  Since this is the address of the site, it is the most logical place on the screen to go to, if in doubt.

  • Anonymous
    September 28, 2006
    The comment has been removed

  • Anonymous
    September 28, 2006
    Microsoft sponsors an antiphishing technology bake-off. Guess who wins...

  • Anonymous
    September 28, 2006
    The comment has been removed

  • Anonymous
    September 28, 2006
    Steve: we didn't include Firefox version 2 because at the time our test started it wasn't available in beta. Because 2.0 incorporats the same code base as Google's Safe Browsing add-on for Firefox 1.5. I think the results are representative, although I'm sure you're right that the release version of 2.0 will do a better job.

    Shane: I wrote you a lengthy mail explaining why we thought SiteAdvisor was an anti-phishing tool. Just in case your mail filtering system blocked it, I've explained our reasoning at <http://www.robichaux.net/blog/2006/09/mcafee_siteadvisor_sure_looks_like_an_an.php>. If you'd like to discuss this further, my door is always open.

  • Anonymous
    September 28, 2006
    "Why in the study, was there no comparison against Firefox 2.0 Beta X or RC1"

    I don't see how a minor version update matters in the results.

    I still don't understand which feature made it require a major version increment.

    Sounds to me like they're worried about IE's major version increment.

  • Anonymous
    September 28, 2006
    "we didn't include Firefox version 2 because at the time our test started it wasn't available in beta."

    A valid explanation, but ultimately futile. In no time some other reason to distrust this study will be brought up, simply because to some, studies that have MS coming out on top are automatically 'suspicious.'

  • Anonymous
    September 28, 2006
    Wow, looks like the firefox lobby has arrived. I guess this story was posted on slashdot?

  • Anonymous
    September 28, 2006
    @ Jill

    You said: "As an avid user of the Internet for many years, it would take a heck-uv-a-lot of a scam, to actually get me to part way with my money, or be convinced that the site was legitimate."

    That may be correct for me and you (let's call ourselves 'power users' - not sure if you would brand yourself as that, but I would, and I'll assume you are fairly competent computer user).

    However lets take my mum... or my Gran.. or heck my 25 year old friend - varying degrees of computer literacy, but would all probably fool for a phishing scam if it was good enough.

    I agree some are bad... no, in fact, some are really bad - but some are also good. Came across one in my mum's inbox 'from PayPal'... HTML email was well crafted, good English, website looked very authentic. The link text in the email was different from the actual URL - but not everyone would check.

    Suppose all i'm saying is this feature isn't for you and me... its for your mum, dad, grand, aunt, or anyone else you know who isn't a fairly knowledgeable web user.

  • Anonymous
    September 28, 2006
    Hey, you, YES YOU, firefox fan boys, please don't post off-topic posts. If you don't like IE just don't go there. JUST DON'T.

  • Anonymous
    September 28, 2006
    Hrm.... Seems to have done the best job of blocking what it detected, where you get 2X the points than for just detecting and warning users. When it comes to warning, IE7 didn't win in any category.

    http://weblog.infoworld.com/techwatch/archives/008114.html

    Great work so far, but I'd like to see better performance in detection.

  • Anonymous
    September 28, 2006
    I don't know about Microsoft Phishing Filter, but I think Google Safe Browsing gets better the more users participate. Thus speaking of accuracy, I could imagine a boost once Firefox 2 ships.


    > I still don't understand which feature made it require a major version increment.
    > Sounds to me like they're worried about IE's major version increment.

    Let's be fair. Firefox has been a decent browser since 2004, whereas Internet Explorer wasn't. It made a straight progress with 1.5, whereas Internet Explorer didn't. Now it's no surprise that Mozilla doesn't have to fix as much as MS. If you want a comparison, then take IE 5.5 and 6.

  • Anonymous
    September 28, 2006
    Anti Phishing keeps picking on Yahoo! Answers every time I try to log on

  • Anonymous
    September 28, 2006
    Jill,

    From February to Mid Aug 2006 the Phishing Filter helped block over 800,000 instances of people trying to access reported phishing websites using IE7 or MSN/Windows Live Toolbar.  This figure includes almost 500,000 blocks since IE7 Beta 2 was released.

    IE7 users are reporting up to 4,500 potential phishing sites per week.

    Microsoft has been adding up to 17,000 URLS a month to its Phishing Filter service.

  • Anonymous
    September 28, 2006
    Unfortuantly, anything that shows Microsoft came out at the top will immediatly be looked upon with suspicion because of the well known sucurity problems with Internet Explorer 6 and Windows XP. Microsoft will have to work hard to turn that negative image around.

  • Anonymous
    September 28, 2006
    Unfortuantly, anything that shows Microsoft came out at the top will immediatly be looked upon with suspicion because of the well known sucurity problems with Internet Explorer 6 and Windows XP. Microsoft will have to work hard to turn that negative image around.

  • Anonymous
    September 28, 2006
    How beta testing of IE7 on Connect goes with regards to the filter:

    q> IE7 crashes on these websites: {...}
    A> Do you have phishing filter on?
    q> yes
    A> Turn it off. Closed

  • Anonymous
    September 28, 2006
    Read the Paul's blog entry about McAfee http://www.robichaux.net/blog/2006/09/mcafee_siteadvisor_sure_looks_like_an_an.php

  • Anonymous
    September 28, 2006
    @Steve
    Firefox 2.0 uses Google antiphishing

  • Anonymous
    September 29, 2006
    Can i test myself the antiphishing filter? Like the EICAR tests? Any URL known to be malicious?

  • Anonymous
    September 29, 2006
    I've been using Sitehound, a product of Firetrust, a NZ company.  Sitehound uses an internally stored URL database, updated daily.  I turned it off and clicked a few banners that are known scam/spyware sites.  The antiphishing filter let me into about half of them.  If addition, the antiphishing filter gave me several false positives, including my own Comcast web mail beta email account.  While I don't have any measurable data to support this, my browser seemed sluggish with antiphishing filter engaged.  So I disabled it and enabled my Sitehound again.  I'm not intending to bad-mouth IE7.  I love it.  This is just feedback FYI.

  • Anonymous
    September 29, 2006
    Does Opera or Safari have or will have this feature as IE7 and Firefox 2 will? Good to see this being added. Any idea when we'll get a slightly newer build then 7.0.5700.6?

  • Anonymous
    September 29, 2006
    it's a challenge for you guys (IE Team) to make this broswer the best of all,please

    It's important you keep improving, adding new functionality prior to release

  • Anonymous
    September 29, 2006
    @Schreuder: See http://www.antiphishing.org/phishing_archive/phishing_archive.html for known phishing sites.

  • Anonymous
    September 29, 2006
    Shane Keats,

    As I said in response to your comment on my blog, your service warns of "fraudulent practices" and has tested "sites representing more than 95% of worldwide Web traffic" and performs "tens of thousands" of tests every day, but phishing sites aren't included?

    http://www.siteadvisor.com/download/ie_learnmore.html

    No exclusion of phishing sites here either:

    http://www.siteadvisor.com/press/faqs.html#q11

    Perhaps you should be more specific about what these "fraudulent practices" are (fraud, but not phishing, despite phishing being a type of fraud?) and add a mention about not covering phishing in the FAQ in addition to the Support Centre (people won't go to the support centre unless they have problems).

  • Anonymous
    September 30, 2006
    PingBack from http://www.teklow.com/weblog/2006/09/30/ms-anti-phishing-tool-wins-ms-bakeoff/

  • Anonymous
    September 30, 2006
    Il est amusant de constater que les simples faits qu’une &#233;tude ait &#233;t&#233; financ&#233;e par Microsoft et que...

  • Anonymous
    October 03, 2006
    PingBack from http://www.aesjkt.com/wp2/?p=540

  • Anonymous
    October 04, 2006
    The comment has been removed

  • Anonymous
    October 04, 2006
    @George  You seem very pro-Microsoft in your comments.  Thats good.  Out of interest, you say you use MS products now/and intend to in the future, also good.  But one question.  Have you tried other Web browsers? e.g. Opera, Maxthon, Firefox? I would hazzard to guess you haven't, since I think you'll find the other products even more impressive. Even the "tabs" are better in IE6, with the MSN Toolbar. (((Yes MS Developers... that's right! The tabs in IE6, with the MSN Toolbar, are BETTER than the tabs in IE7)))

  • Anonymous
    October 04, 2006
    @Harold: I'd love to hear more about what you think is better about the MSN Toolbar's tab implementation.  Thanks!

  • Anonymous
    October 06, 2006
    actually IE7 tabs are much better than MSN Toolbar

  • Anonymous
    October 09, 2006
    PingBack from http://www.itechnote.com/2006/10/09/anti-phishing-impacts-on-web-site-owner/

  • Anonymous
    October 10, 2006
    It is nice to have  Anti-Phishing  Services available at individual Enterprise level. As I understood that if  Anti-Phishing  options enabled , then every link will be scanned by "Anti-Phishing  services or by servers at microsoft.com " . This is kind of privacy concern. If Microsoft come up having tool to have Anti-Phishing  service or server available at individual enterprises ( like the way SUS is) , then individual enterprises sync with Microsoft Servers and route all the traffic thru  enterprise owned and controlled  Anti-Phishing  Services/Servers.

  • Anonymous
    November 14, 2006
    Some of you may have seen stories comparing IE7’s anti-phishing accuracy with our competitors, citing

  • Anonymous
    November 18, 2006
    PingBack from http://adelinos.wordpress.com/2006/11/18/firefox-e-mais-confiavel-que-o-ie7-contra-phishings/

  • Anonymous
    November 19, 2006
    PingBack from http://shahrul.info/blog/?p=97

  • Anonymous
    November 21, 2006
    McAfee, which originally disputed SiteAdvisor's inclusion in the 3Sharp phishing filter tests back in

  • Anonymous
    November 28, 2006
    PingBack from http://accuracy.pierresccservice.com/anti-phishing-accuracy-study/

  • Anonymous
    December 07, 2006
    PingBack from http://accuracy.newstack.com/anti-phishing-accuracy-study-2/

  • Anonymous
    January 16, 2007
    Paul Thurrott, in his somewhat late review, takes a look at Firefox 2's new features. Unfortunately, and something I didn't expect from Paul, it's either the most intentionally misleading review of Firefox, or it's a completely unserious piece of writing.

  • Anonymous
    February 06, 2007
    Back in November, we announced our intention to bring Extended Validation SSL Certificates to IE7 . This

  • Anonymous
    February 06, 2007
    PingBack from http://computer.chardum.com/uncategorized/2007/02/06/ie7-at-rsa-san-francisco/

  • Anonymous
    March 13, 2007
    PingBack from http://winblogs.security-feed.com/2006/09/28/anti-phishing-accuracy-study/

  • Anonymous
    December 14, 2007
    PingBack from http://mozillafirefox3.net/firefox-myths/

  • Anonymous
    May 29, 2009
    PingBack from http://paidsurveyshub.info/story.php?title=ieblog-anti-phishing-accuracy-study

  • Anonymous
    June 09, 2009
    PingBack from http://greenteafatburner.info/story.php?id=3455

  • Anonymous
    June 15, 2009
    PingBack from http://einternetmarketingtools.info/story.php?id=7644