Johnvey.com - Purveyor of Fine Words » Blog Archive » Concise Adblock Filter Set Explained

January 14, 2007

January 14 2007

Concise Adblock Filter Set Explained

Adblock is the single most useful Firefox plugin available today. Just like watching sitcoms with automatic commercial-skip, adblock’s banner ad supression system elicits a smug sense of satisfaction even after browsing through your 10,000th ad-free web page. However, a huge barrier to adoption seems to be the lack of a default filter set, so when you first install adblock, nothing happens.

The main issue is that adblock does not have any intelligence as to the content that is included with a webpage; it is just a generic regex-based filter system, so it is only as effective as the filters that you provide. There are plenty of pre-made lists available but they tend to be overly-aggressive in what is supressed, resulting in occasional broken pages and/or pages that dead-end because adblock has removed the “Next” button. The most dangerous public set seems to be the EasyList, which has a 360+ item block list. Evidence that the creators know of its greedy nature is their inclusion of a 20+ item whitelist to manually compensate what was initially blocked. Even more unstable is the EasyElement list that searches through the DOM to remove suspected elements directly from the main document — a list of 570+ substrings to search for.

Intead of using such a large, reactive list of simple and site-specific string matches that tries to supress 100% of ads, I posit that you only need 2 adblock filters to eliminate 70-80% of ads, and still be confident that legitimate content isn’t being flagged as a false positive. By getting into the heads of HTML writers, we can pick out the most common patterns used to include ads and create regex patterns to suppress the ads.

/(\b|_)ad(x|s?)(\b|_)/
This regex looks for any element that contains the string ‘ad’, ‘ads’, or ‘adx’ surrounded by a word boundary, because the vast majority of web sites partition their ads into a single directory or serve them through a single script. The word boundary check is crucial to this filter because just searching for the characters ‘ad’ is ineffective. Instead, the word boundary restriction means that adblock will supress elements that contain strings like ‘ads.server.com’ or ‘www.server.com/ads/’ or ’server.com/ad_server.php’, but not ‘adobe.com’ or ’server.com/adjustment’.
/ad.*\d+[xX]\d+/
This regex exploits the common technique of ad designers to use the image dimensions in their element name, i.e., “server.com/newads.php?location=top&size=468×80″. Like the previous rule, we don’t just exclude any element that has dimensions, but qualify that by searching for the string ‘ad’ as well.

At this point, your browsing experience will be significantly improved, but you can bump up your block rate to about 80-90% with a few more simple substring matches. There are many well known ad providers that exist solely to deliver ads, so we can consildate those in composite filter rules:

/a(2\.yimg|dserv|dvert|tdmt|twola)/
This rule collects all the ad serving systems that start with ‘a’: Yahoo, Atlas, AOLTimeWarner, and generic ad serving systems.
/b(anners|logads)/ falkag.net
These pick up anything labeled with ‘banner’, the ‘blogads’ network, or Falk AdSolutions.

Realistically, reducing the ad load by 90% should be more than sufficient for anyone. Chasing that last 10% — and whitelisting the collateral damage — will always be a losing battle. Your time is better used reading the content that is on the page you requested in the first place.

3 Comments »

Funny how you measure the “dangerousness” of a list by the number of filters. Any reason why you would say that http://adblock.free.fr/adblock.txt is less dangerous? It has less filters but those filters are so complex that hardly anybody can tell what they block.

Easylist goes by the recommendations for Adblock Plus - use specific filters and avoid regular expressions. This allows the filter list to be processed very fast. And the whitelisting entries are mostly due to the fact that some sites started to serve regular content through known advertising sites.

Have a look at the Filterset.G whitelist (http://pierceive.com/filtersetg/whitelist-beta/) - now that’s scary…

Comment by Wladimir Palant — January 15, 2007 @ 3:14 pm
“Dangerous”? That’s a bit harsh don’t you think?

The EasyList is fairly aggressive because users want it that way … and “yes” there will be an occasional ‘burp’ in what it blocks. But considering the amount of users the EasyList has, problems have been very minimal at best and false-positives have been addressed NOT mainly through whitelisting, but rather through a rewrite of the filtering strings.

You wrote:
“Evidence that the creators know of its greedy nature is their inclusion of a 20+ item whitelist to manually compensate what was initially blocked.”

The irony of your statement is that your first proposed filter string:
/(\b|_)ad(x|s?)(\b|_)/
…is EXACTLY why about 80% of those whitelist strings exist. Most of the whitelistings are for video players served thru an “ad” string. The whitelists allow the player to function correctly on some very MAJOR sites without having to remove the broader generic filter strings like */ads/* or *//ads.*. And I don’t have to whitelist the ENTIRE page. The EasyList works quite well this way. Try watching news video at FoxNews, MSN, Forbes, etc with just that one filter string that you proposed …. you will have a whitelist larger than mine with just that one string.

You wrote:
” … resulting in occasional broken pages and/or pages that dead-end because adblock has removed the “Next” button.”

I don’t know what ‘next’ button you are talking about. Things like this could occasionally happen, but I currently have no reports of any big problem with things like that … and if someone did have a problem, I would hope that they would bring it to my attention. These things are usually fixed as fast as I can fix them when they occur. Trying to keep pages free of ads without interrupting a user’s surfing experience is no small task … but I love doing it and devote a lot of my time to it. :-)

ps: Adblock Plus does NOT have a problem with large filter lists as long as they follow the simple expression ’shortcut’ rules. So using string totals is irrelevant to ABP’s operation. I increased the filter size because it does not take any noticeable performance hit.

Sincerely:
rick752 - ABP EasyList/EasyElement author.

Comment by rick752 — January 15, 2007 @ 5:18 pm
I’m not sure who you’re writing for. Are you targetting end-users or filter subscription maintainers? If the former, then they’re not spending time figuring out the best way to create filters. If the latter, then EasyList is the way it is because it’s optimized for the way Adblock Plus works. You can read all about it on adblockplus.org where excellent documentation is maintained and Wladimir explains in his blog which filters work best. I think you’re still used to Adblock’s filter style where regular expressions are preferred and people try to cram as many rules as they can in one expression. That is bad form for Adblock Plus because those types of filters are slower and make debugging harder.

Comment by Stupid Head — April 20, 2007 @ 1:36 pm

RSS feed for comments on this post. TrackBack URI

Reading

Viewing

Linking

I wish this map of current US gas prices factored out the taxes included in the pump price. It seems like what the map mostly shows is the differences in taxes between states (PDF map) and not, for instance, how the distance from shipping ports or local demand affects prices. (via what i learned today)
(link)
Fractal furniture!

One could imagine a Powers of Ten video with drawer pulling instead of zooming.
(link)
[Part four of a recurring series...part one, part two, part three.]

According to the Indian National Crime Bureau, there were 6,787 dowry deaths in India in 2005. A dowry death occurs when a woman is killed or commits suicide due to coercion by her husband or her husband's family in order to secure a larger dowry. [Indian National Crime Bureau]

As of August 2005, the poverty rate in Mississippi was 21.1%, the highest in the nation. The state also ranks first in senior poverty and second in child poverty. Despite being surrounded by states with relatively low poverty rates, Washington DC ranks first in child poverty and is second in overall and senior poverty. [USCCB]

According to the Zoological Society of London, between a quarter and a third of the world's wildlife has been lost since 1970. [BBC]

Buddhist teachers Michael Roach and Christie McNally haven't been more than 15 feet from each other in the ten years since they took an oath to that effect. They also read the same books at the same time. [NY Times]

There are more Chinese restaurants in the US than McDonald's, Burger King, KFC, and Wendy's restaurants combined. [YouTube]

NYC's alternate-side parking rules will be suspended in Park Slope for a few months so that workers can replace parking signs. Residents are overjoyed because they don't have to move their cars every few days. [NY Times]

There are at least 3 escalators in Wyoming. [Metafilter]

Velcro is 50 years old. (At least the trademark is.) [mental_floss]

The Golden Gate Bridge is younger than John McCain. [Things Younger Than John McCain]
Oklahoma City is repairing the state's busiest highway by tearing it down and building a park in its stead.

In Oklahoma City, the interstate will be moved five blocks from downtown to an old railroad line. The new 10-lane highway, expected to carry 120,000 vehicles daily, will be placed in a trench so deep that city streets can run atop it, as if the highway weren't there. The old highway will be converted into a tree-lined boulevard city officials hope will become Oklahoma City's marquee street.

Several other cities have done (or are planning to do) similar highway tear downs.

"Highways don't belong in cities. Period," says John Norquist, who was mayor of Milwaukee when it closed a highway. "Europe didn't do it. America did. And our cities have paid the price."

No mention of Boston's Big Dig, perhaps the most high-profile example of this trend.
(link)
The Wii Fit, the new exercise peripheral for the Nintendo Wii, was reviewed favorably by a number of people for the New York Times. A fitness professional at the Sports Center at Chelsea Piers gave it pretty high marks:

"Actually I think it's pretty good," she said. "You can definitely get a workout. When I started doing it, I realized all the activities were pretty much on point. There were some things I didn't like, like the alignment in a couple of places. But over all, I thought they did a good job and this will be a good tool for people who can't make it to the gym."

The Wii Fit will be released in the US and Canada early next week.

Update: Joel Johnson has a nice round-up of exercise-themed video game accessories, from the unreleased Atari Puffer to the Wii Fit.
(link)
A list of ways to get yourself excused from the jury pool in the R. Kelly child pornography case.

I (heart) R. Kelly. Nothing gets prospective jurors booted faster than telling the prosecution they are a fan of Kelly's. Just ask the woman who called him a "musical genius." When prodded to say something negative about Kelly, the best she could come up with was: "He and [rapper] Jay-Z don't get along?" Prosecutors bounced her soon after.

Another potential juror was excused for suggesting that Kelly "led the Taliban in attacking us on 9-11".
(link)
A collection of photos of a cleaning crew washing Seattle's Space Needle with high pressure washers (scroll down a bit).

Even though the sprayers use half the flow of a garden hose, the water shoots out at 3,000 pounds per square inch -- more than enough power to send the guy behind the hose flying. "One thing we say is, it doesn't necessarily have to be fun to be fun. There are definitely times when I'm spinning in free space and I'm like, holy cow this is terrifying and I can't believe this is my job," said Matt Henry, rope technician.

The company doing the work, Karcher GmbH & Co., has done similar high-profile jobs around the world, all at no cost...their web site says that these projects are good publicity for their cleaning products. Here's a sampling of some other projects they've done, including the Statue of Liberty and Berlin's Brandenburg Gate. (via girlhacker)
(link)
Here's an update on the effort to solve the Pioneer anomaly, the unexplained deviation in motion of deep space probes from what Newton and Einstein's theories predict.

As it sped through space, a specialist in radio-wave physics named John Anderson at NASA's Jet Propulsion Laboratory noticed an odd thing. The spacecraft was drifting off course. The discrepancy was less than a few hundred-millionths of an inch per second for every second of spaceflight, accumulating year after year across billions of miles. Then Pioneer 11, an identical probe escaping the solar system in the opposite direction, also started to veer off course at the same rate.
Ordinarily, such small deviations might be overlooked, but not by Dr. Anderson. He monitored the trajectories six years before calling attention to the matter. "I'm a little like an accountant," Dr. Anderson said. "We have Newton's theory and Einstein's theory, and when you apply them to something like this -- and it doesn't add up -- it bothers me."

The researchers, using data recovered from recently discovered Pioneer records and funded by sources outside of NASA, have figured out part of the problem but the rest remains a mystery.
(link)
Greg Allen still has his bottle of Suck Cola from when the now-defunct web site Suck was handing them out at a trade show in 1996. He's building a registry of Suck Cola bottles...if you've got one, send in the details.

After your Cola information is reviewed and validated, you will be issued a Suck Cola Registry Number. I have designated my bottle SC0005, having reserved the first four Registry Numbers, SC0001-SC0004, for Suck.com co-founders Joey Anuff and Carl Steadman.

Suck the web site has now been dead for as long as it was active, but the Cola lives on.
(link)
Video of designer John Gall, who shares his five rules for book cover design.

The other great source of inspiration is the deadline.
(link)
A look at how portion sizes have changed in the US over the years.

We don't have to eat those extra 360 calories in the tub of popcorn, but that's easier said than (not) done. Studies indicate that when given food in larger containers, people will consume more. In a 1996 Cornell University study, people in a movie theater ate from either medium (120g) or large (240g) buckets of popcorn, then divided into two groups based on whether they liked the taste of the popcorn. The results: people with the large size ate more than those with the medium size, regardless of how participants rated the taste of the popcorn.
(link)

Links provided by kottke.org.

Offering

Syndicating

Feedburner RSS

2001–2005 A brief exploration through the marvelous mind of Johnvey Hwang. All content released under a Creative Commons License unless otherwise noted.

PURVEYOR OF FINE WORDS