Ignoring user agents

The next idea was to ignore clickthroughs from all user agents (browsers) except IE, Netscape and Opera. Again, I let them clickthrough, just don't make the advertiser pay for them.

The code below shows how this was done:

// ignore any user agent that isn't Mozilla (IE and Netscape) or Opera var sAgent = '' + Request.ServerVariables ( 'HTTP_USER_AGENT' ); // make lowercase sAgent = sAgent.toLowerCase ( ); // should we count this agent? if ( -1 != sAgent.indexOf ( 'mozilla' ) || -1 != sAgent.indexOf ( 'opera' ) ) {    // it's an unknown user-agent    bIgnoreClick = true; }

It was suggested that I also ignored known IP addresses used by spiders. I found an excellent source at Search Engine World that documents these.

After investigation however, I noticed that all major spiders except one, Lycos, used very unique user agents. They would all get caught by the user agent check above!

Lycos was an exception - their spider sometimes masqueraded as IE 5.0. Luckily, they also have "Lycos_Spider" in the agent string, so I modified the test above to cope with that too:

// should we count this agent? var bKnownBrowser = ( -1 != sAgent.indexOf ( 'mozilla' ) || -1 != sAgent.indexOf ( 'opera' ) ); if ( -1 != sAgent.indexOf ( 'spider' ) )      // lycos spider acts like mozilla    bKnownBrowser = false; if ( !bKnownBrowser ) {    // it's an unknown user-agent, so ignore them    bIgnoreClick = true; }

Next, Ignoring HTTP commands...