Suresh is postmaster for Outblaze.com, which manages email for tens of
millions of Internet users.
Danny is EFF's activist coordinator and is involved in setting up the
DearAOL.com project.
I think I captured the entirety of their exchange. Probably best to read
it from the top down.
Previous Politech messages:
http://www.politechbot.com/2006/04/15/details-on-how/
http://www.politechbot.com/2006/04/13/why-was-moveonorg/
http://www.politechbot.com/2006/04/15/john-gilmore-on/
-Declan
-------- Original Message --------
Subject: Re: [IP] Why was Moveon.org blocked by AOL? Did recipients want
the email messages? [sp]
Date: Fri, 14 Apr 2006 16:39:34 +0530 (IST)
From: Suresh Ramasubramanian <suresh@private>
To: dave@private, suresh@private
CC: danny@private, declan@private
References: <20060414082600.GA16537@private>
<BE049889-B2CB-43EF-885A-6E55AB794C5B@private>
On Fri, April 14, 2006 15:26, David Farber wrote:
>
> From: "Danny O'Brien" <danny@private>
> I think I can answer these questions.
>
> No-one receives any mail from any dearaol.com address. I run a
> Mailman list
The dearaol.com URL is / was being blocked by AOL as it turns out.
My reference was to the "send this message to a friend" function on the
dearaol.com site, and which, like all send to a friend functions, is prone
to misuse. Try this experiment. Go to the dearaol.com site, and "sign the
dearaol.com petition as an individual", giving information there like
"my@private" (which by the way is a spamtrap on the domain email.com,
which does exist). You are then presented with a page that asks you to
enter the email addresses of a bunch of your friends, to whom you can pass
on the message about dearaol.com. No way is that ever going to be
confirmed optin or anything at all like it.
Plus several even worse managed newsletters that'd refer to this campaign.
Whatever the source, you'd find a whole lot of email getting reported as
spam, that has the dearaol.com URL in it.
> (And I should point out to Suresh that if he believes lists that aren't
> confirmed double opt-in are spam, he's going to have serious problems
with
No, Danny - I dont believe that they are spam. But I *know* (forget
"believe") that they have a greater than normal chance of getting a high
number of bad addresses into them - though in practice, for most lists,
that's not a very big problem. However, combine that with a "Tell all
your friends about this" pages on campaign sites and you get something
that's far better at generating irritating chain mail than it is 0at
spreading memes.
> Somehow - and this is what AOL's tech support folk told me when I
> called them this morning - they identified www.dearaol.com as a
"morpher".
> This is a site that redirects user clicks to many different sites.
> It's true: www.dearaol.com has round-robin DNS. I plead guilty to
> load-balancing of the most heinous kind.
Ah. Multiple IP addresses mapped to a single A record - that's round
robin DNS. The 1 minute TTL (time to live) part of that A record is what
makes it hard for a filter that's exclusively machine based to distinguish
it from a site that's hosted on "botnet" virus infected PCs and sets
multiple hosts with short TTLs just so that the spam site can skip from
one virus infected PC to another at a moment's notice.
What you get with those virus hosted sites is somethign that looks very
similar only they're mapped to compromised hosts of one type or the other.
example.com. 1M IN A infected.pc.on.broadband
example.com. 1M IN A cracked.linux.1u.pizzabox
[tech wise - You really need 1 minute instead of (say) 1 hour TTLs,
though? Because that's not a very foolproof way of load balancing due to
the way say bind works ... you'd be advised to use something like a L2
load balancing switch, or perhaps http://w.ods.org/haproxy/ ..]
And when you analyze email that gets reported as spam, for URLs that have
such attributes, you are likely to pick up dearaol.com along with 300 or
400 randomly created spammer domains every day. So, anybody doing that
sort of thing probably has a review process, and removes false positives
as and when they crop up (false positives defined as catching legit sites
that get reported in spam, as opposed to what you really want to catch,
zombie hosted spammer domains)
> AOL appears to have taken this as a sure-fire indication of a
> spamming site, and instantly banned *every email that mentions this URL*
> from entering the AOL system.
Well, I understand the way their thought processes works .. and I've seen
hundreds of sites matching that pattern crop up every day, AND is a URL
that shows up in email that gets reported as spam by your users, along
with thousands of such spam reports a day.
> working days. EFF has received reports of these kind of URL bans before.
> Bennett Hasselton, of the free speech group PeaceFire, has documented
> many innocent groups who find all mails discussing their URLs removed
from
> AOLspace.
One way to go - plug that URL (or at least the parent URL in cases like a
free webhosting domain) into http://groups.google.com along with the word
"spam". See what you get. Its not unknown that a free webhosting site can
have a problem with spammers signing up thousands of accounts on their
site, and then setting up all those accounts to redirect to the
"mothership" spam site - typically child porn or warez, in previous cases
I've seeen, so does generate a higher than usual volume of complaints.
When that happens, its quite feasible that a provider may elect to block
all email containing the spammed free webhost (as faster than individually
blocking hundreds of spammer created free account URLs on that host, and
as a motivator for that host to clean up its act, fast).
At least, that explains the example Bennet gave me when he emailed me in
response to my politech post.
> or put out a press release, www.dearaol.com would still be banned from 20
> million user's private communications, and would remain so until I made
> that call.
AOL has something rather more than just 20 million mailboxes.
> I'm more disturbed that Suresh had a similiar block, which he finally
> deigned to remove because he believed us to have "legitimate" popularity.
dearaol.com turned up in email that was reported as spam.
Our filters picked it up
We have a manual review process that regularly reviews domains that are
snagged by our filters - at least once a day. And any domains that stand
out as false positives get weeded out of our filters .. (again, that means
something that's not an obvious "bad spam" site .. anything that has the
potential to appear in "ham" / non spam email - say an astroturf activism
site - does get exempted, and fast).
We return a clear spamblock.outblaze.com URL that's tagged / unique for
every error, to make it easier for the sender, and for us, to find out
what's blocked, and for any false positives to be addressed faster. And
the postmaster staff - my team - replies ASAP, within a few hours or at
least within a day, to such emails.
> Suresh's
> company manages filters for over 40 million users. I'm happy that Suresh
> likes me enough personally to let me escape his blacklists, but when
"Not that I disliked dearaol.com's methods less, but that I disliked spam
more", to paraphrase Brutus' eulogy of Julius Caesar. [I always preferred
that to Mark Antony's rabble rousing speech that followed it .. ]
We try hard not to apply filters that block valid emails. And we actively
remove any false positives, both where we notice them, and where they're
brought to our attention.
Oh, and I must again stress that we do appreciate reports of false
positives - preferably direct to our postmaster staff instead of third
hand from reporters fed a frantic story about censorship.
--srs
----- Original Message -----
From: "Danny O'Brien"
To: "Suresh Ramasubramanian"
Subject: Re: [IP] Why was Moveon.org blocked by AOL? Did recipients want
the email messages? [sp]
Date: Fri, 14 Apr 2006 12:47:08 -0700
On Fri, Apr 14, 2006 at 04:39:34PM +0530, Suresh Ramasubramanian wrote:
> On Fri, April 14, 2006 15:26, David Farber wrote:
> >
> > From: "Danny O'Brien"
> > I think I can answer these questions.
> >
> > No-one receives any mail from any dearaol.com address. I run a
> > Mailman list
On 4/14/06, Suresh Ramasubramanian wrote:
> On Fri, April 14, 2006 15:26, David Farber wrote:
> >
> > From: "Danny O'Brien"
> > I think I can answer these questions.
> >
> > No-one receives any mail from any dearaol.com address. I run a
> > Mailman list
[genuinely fascinating insights into filtering - thanks, Suresh]
>
> We return a clear spamblock.outblaze.com URL that's tagged / unique for
> every error, to make it easier for the sender, and for us, to find out
> what's blocked, and for any false positives to be addressed faster. And
> the postmaster staff - my team - replies ASAP, within a few hours or at
> least within a day, to such emails.
This is a classic example of why senders and receivers need to work together
on these issues, and why I worry that the revenue share aspect of AOL's deal
places them as contractual adversaries. One of the greatest collateral
effects
of spam has been that bounces of any kind have been discredited (because of
joe-jobs and misdirected bounces), and mailing list software has
traditionally
eaten such data without relaying it to mailmasters. I'm sure someone is
already considering this, and I'm always a little loathe to introduce new
ideas into an already intensely-debated feed, but perhaps a feedback tag in
DKIM would make everybody's life easier in providing these notifications --
especially on domain-level bans like this. Feedback loops seem to be
something
that everyone would value.
> > Suresh's
> > company manages filters for over 40 million users. I'm happy that Suresh
> > likes me enough personally to let me escape his blacklists, but when
>
> "Not that I disliked dearaol.com's methods less, but that I disliked spam
> more", to paraphrase Brutus' eulogy of Julius Caesar. [I always preferred
> that to Mark Antony's rabble rousing speech that followed it .. ]
>
I worry exactly who you're trying to assassinate here :)
d.
>
> --srs
>
>
-- suresh ramasubramanian suresh@private gpg # EDEDEFB9
manager, security & antispam operations, outblaze limited
-------- Original Message --------
Subject: Re: [IP] Why was Moveon.org blocked by AOL? Did recipients want
the email messages? [sp]
Date: Sat, 15 Apr 2006 08:48:18 +0530
From: Suresh Ramasubramanian <suresh@private>
To: danny@private, "Suresh Ramasubramanian" <suresh@private>
CC: dave@private, suresh@private, declan@private
Hi
Just two points to add. AOL pioneered the idea of feedback loops, and we
were probably the second ISP to put them into production on a large scale.
http://postmaster.info.aol.com/fbl/
We're (along with a bunch of other ISPs - aol, yahoo etc) working on ways
to make these feedback loops automated and machine parseable to some extent
by putting them into the standard ARF format (www.mipassoc.org/arf/). We
and AOL (as well as some other ISPs - roadrunner and earthlink, recently)
offer feedback loops that are ARFd, currently.
If you have block concerns a very good way to go would be to setup a
feedback loop, monitor complaint rates for yourself, and see if you cant
unsub complainers fast (or add them to a suppression list so that they cant
receive dearaol alerts, for example). Spamtrap hits are of course a
special case that wouldnt really be covered by feedback loops and may lead
to blocks - but well, any practice at all that helps avoid mailing to
spamtraps is a good practice (and that translaes to switching everything to
confirmed optin, and putting some limits in place on the send to a friend
part of any sites you run .. oh, and some way to encourage responsible
publicity of ideas when someone else runs a mailing campaign referncing or
using any of your resources.
In any case - what happened at aol seems to have been a honest false
positive as opposed to any move to censor you
And as for email - why, that's the difference between filtering and (say)
assasination. I or my guys here can turn on and off filters with a single
mouse click. That's a particularly significant power when its exercised
over a user base our size (and aol has more than triple the number of our
users).
So you typically dont find people who are stupid and power mad enough to
think spam filtering is a mandate for them to block anything they want
just because they dont like the content and want to censor it. There's
an automatic tendency to be moderate and balanced, because we're finally
answerable to our users. Several people from the EFF just dont seem to
realize that, at least going by various eff deeplinks and press releases
I've read in the past.
That is probably why you'll find several very dismissive emails and
articles from me about EFF press releases that talk about spam.
Describing everything you see in the spam filtering world as based on an
inherent desire to censor is factually wrong.
regards
srs
-------- Original Message --------
Subject: Re: [IP] Why was Moveon.org blocked by AOL? Did recipients want
the email messages? [sp]
Date: Fri, 14 Apr 2006 21:44:24 -0700
From: Danny O'Brien <danny@private>
Reply-To: danny@private
To: Suresh Ramasubramanian <suresh@private>
CC: Suresh Ramasubramanian <suresh@private>, dave@private,
declan@private
References: <20060415031818.13757.qmail@private>
On Sat, Apr 15, 2006 at 08:48:18AM +0530, Suresh Ramasubramanian wrote:
> Hi
>
> Just two points to add. AOL pioneered the idea of feedback loops, and we
> were probably the second ISP to put them into production on a large
scale.
>
> http://postmaster.info.aol.com/fbl/
>
> We're (along with a bunch of other ISPs - aol, yahoo etc) working on ways
> to make these feedback loops automated and machine parseable to some
extent
> by putting them into the standard ARF format (www.mipassoc.org/arf/). We
> and AOL (as well as some other ISPs - roadrunner and earthlink, recently)
> offer feedback loops that are ARFd, currently.
>
I knew about ARF (and one of the first steps I performed when we saw the
bouncing was to sign up for an AOL feedback loop, something that we haven't
needed until now). The elaboration I was grappling for was providing a
feedback loop for domains, as opposed to IPs, to deal with URL blacklisting.
As I say, I bow to ongoing developments in this area.
> If you have block concerns a very good way to go would be to setup a
> feedback loop, monitor complaint rates for yourself, and see if you cant
> unsub complainers fast (or add them to a suppression list so that
they cant
> receive dearaol alerts, for example). Spamtrap hits are of course a
> special case that wouldnt really be covered by feedback loops and may
lead
> to blocks - but well, any practice at all that helps avoid mailing to
> spamtraps is a good practice (and that translaes to switching
everything to
> confirmed optin, and putting some limits in place on the send to a friend
> part of any sites you run .. oh, and some way to encourage responsible
> publicity of ideas when someone else runs a mailing campaign
referncing or
> using any of your resources.
>
And they say senders bear none of the burden of mail delivery...
> In any case - what happened at aol seems to have been a honest false
> positive as opposed to any move to censor you
I'm of the same opinion: but it's the collateral damage of false positives
that started this whole debate, and the worrying weight of who those false
positives fall upon that has caused us all this concern.
>
> And as for email - why, that's the difference between filtering and (say)
> assasination. I or my guys here can turn on and off filters with a
single
> mouse click. That's a particularly significant power when its exercised
> over a user base our size (and aol has more than triple the number of our
> users).
>
> So you typically dont find people who are stupid and power mad enough to
> think spam filtering is a mandate for them to block anything they
want just
> because they dont like the content and want to censor it. There's an
> automatic tendency to be moderate and balanced, because we're finally
> answerable to our users. Several people from the EFF just dont seem to
> realize that, at least going by various eff deeplinks and press releases
> I've read in the past.
>
I know your feeling Suresh, and I've tried my best to convey that that's not
the sentiment here. I think a lot of the problem arises from the fact that
everyone is trying to do a difficult job to the best of their ability. Our
role in this is to try and highlight unforeseen consequences, and most of
those revolve around pointing out issues that are well-known to people like
yourself, but less known elsewhere.
Most people are rather shocked by the idea that AOL or other ISPs could stop
discussion of a single URL with a click. You have good reasons for doing so,
but the fact remains that those with the power are often very poorly
placed to
determine exactly how moderate they are being, or whether their choices are
wise for other groups. And, of course, the difference between you and
Charles
and others willing to enter the debate, and others who aren't that
interested
in these subtle ethical stances, is vast.
> That is probably why you'll find several very dismissive emails and
articles
> from me about EFF press releases that talk about spam. Describing
> everything you see in the spam filtering world as based on an inherent
> desire to censor is factually wrong.
No aspersion, but I don't think I've seen a censor yet that saw
themselves as
that. It's all good intentions and unintended consequences.
This works both ways of course, which is why it's always good to speak with
you.
d.
>
-------- Original Message --------
Subject: Re: [IP] Why was Moveon.org blocked by AOL? Did recipients want
the email messages? [sp]
Date: Sat, 15 Apr 2006 10:48:53 +0530
From: Suresh Ramasubramanian <suresh@private>
To: danny@private, "Suresh Ramasubramanian" <suresh@private>
CC: Suresh Ramasubramanian <suresh@private>, dave@private,
declan@private
> The elaboration I was grappling for was providing a feedback loop
> for domains, as opposed to IPs, to deal with URL blacklisting.
That's a long standing item on the to do list for this project. Getting
it stabilized and having it keyed off IPs - as is going on now - is just
for starters. But it takes time and spare cycles, and if you can
contribute code and ideas, you'd be welcome.
> And they say senders bear none of the burden of mail delivery...
Well, let's put it this way .. its high time senders started knowing
what they're sending out, and to whom, and doing some due diligence on
both. Putting all the burden on receivers doesnt scale, makes their job
lots more difficult.
Senders working together with receivers tends to produce far better results.
> I'm of the same opinion: but it's the collateral damage of false
positives
Yes, banner headlines that say AOL is censoring the Dearaol coalition
certainly gave me a sense of that (!)
No filtering system is foolproof. The point is that false positives
have to be dealt with responsibly and if at all possible, proactively.
Anybody who filters spam responsibly already understands that.
> everyone is trying to do a difficult job to the best of their
ability. Our
> role in this is to try and highlight unforeseen consequences, and most of
> those revolve around pointing out issues that are well-known to
people like
> yourself, but less known elsewhere.
Hardly "unforeseen", trust me. We realize that mistakes can happen, and
will happen. We rely on people reporting these to us, and we rely on
due diligence and precautions to minimize the occurance of false
positives, and quick removal of any that are spotted.
Press releases that claim this is a deliberate attempt at censorship,
and the use of astroturfing tactics of the sort that are more useful in
a political campaign just aren't the way to go.
I recall giving you contact information for some senior AOL staff (who
arent that hard to find, they're active on most of the antispam
discussion lists), and suggested that You or Brad contact them about it.
I dont know whether you did contact them, but the EFF press releases /
deeplinks so far, and the dearaol campaign's guerilla tactics, are
simply turning this whole thing into a PR circus rather than an
opportunity for reasoned dialogue. But no, all that's being done is a
series of hatchet jobs on AOL.
So, I wonder what the next EFF deeplink would be about -
http://carlhutzler.com/blog/?p=18 say?
> Most people are rather shocked by the idea that AOL or other ISPs could
> stop discussion of a single URL with a click.
Those of us who have that power are quite well placed to determine
exactly how moderate or not we are being. If we block a source of
legitimate email, or a legitimate URL, we'll find out fast thanks to the
complaints that come in.
We're answerable to our users and responsible for delivering email they
ask for. And we're answerable to ourselves not to block valid email
sources. I for one eat my own dogfood, and my personal email is subject
to just the same filters that I apply on our production servers. Ditto
for my team.
(the postmaster mailbox remains completely unfiltered though .. and
thats where spam reports come in).
> determine exactly how moderate they are being, or whether their
choices are
> wise for other groups. And, of course, the difference between you and
Charles
You know, Danny, even the worst spam has at least some people - a
fraction of one percent of the recipients probably - who want to buy the
stuff that's being advertised ("body part enhancement" pills and such).
Badly managed activist mailing lists are a slightly grayer area. But
our spam filters are keyed to complaints, and subject to substantial
oversight so that sites with nonspam potential are unblocked when we see
them, even though in some cases we have to grit our teeth to unblock
because of far higher complaint rates than I've seen moveon at their
worst generate.
> No aspersion, but I don't think I've seen a censor yet that saw
themselves
> as that. It's all good intentions and unintended consequences.
The difference is of course where a censor actively looks for those
consequences and addresses their effects. As we do, and as I'm quite
sure (from personal experience) that AOL does.
Intentions backed with action are always a far better thing than mere
intentions.
-srs
_______________________________________________
Politech mailing list
Archived at http://www.politechbot.com/
Moderated by Declan McCullagh (http://www.mccullagh.org/)
This archive was generated by hypermail 2.1.3 : Sat Apr 15 2006 - 08:21:52 PDT