[liberationtech] Message from Ricken on Avaaz cyberattack

[Hal Roberts]

I'm jumping in here because I think it's important to understand the 
challenges of ddos protection at a more sophisticated level than 
'cloudflare is free!'.

If you are just trying to publish some set of static content, there are 
a variety of methods you can use to do strong ddos protection on the 
cheap.  All of them rely on getting lots of free or cheap bandwidth, 
whether through a big hosting provider like blogger, through a free cdn 
like cloudflare, or through a small human rights oriented protection 
service that subsidizes the bandwidth cost in some way.  That bandwidth 
just helps serve mostly static content, though, and doesn't by itself 
keep an interactive site functional in the face of an attack.

To keep the interactive features of a site (like avaaz.org) up, you have 
to make pretty deep changes in how the site works to be ddos resistant. 
  And that usually involves working with some company or organization 
that is expert in ddos protection.  That means hiring a company like the 
one that avaaz is evidently using (I have no specific knowledge of that 
company, but there is a whole class of companies like it), and they are 
expensive.

And once you are having to embed the ddos protection into the site's 
functionality rather than just its content, it's a lot harder to 
leverage the free sources of content bandwidth.  I'm pretty sure this is 
cloudflare's business model -- providing the simple content bandwidth 
for free but leveraging their (likely justly earned, though I haven't 
tested it) reputation in order to charge for the expertise to protect 
more complex, interactive sites.

When we queried services a couple of years ago for our ddos report, we 
were routinely quoted numbers around $10k a month for protection up to 
10G of traffic.  There are lots of small hosting companies that 
'guarantee' protection up to 1G, but the guarantee is just to get your 
currently monthly bill refunded, hardly what's needed in the face of an 
attack.  And the routine quote of $10k / month was just for the basic 
bandwidth and filtering systems, not including any custom work on the 
interactive parts of the site.

There are certainly human rights oriented individuals and, increasingly, 
smallish organizations who are providing these sorts of ddos protection 
services.  I'm generally supportive of those efforts and know of cases 
in which they have smartly done enormous good.  But those individuals 
and orgs are all subsidized in some way or another, through some 
combination of private and public funding, donations of backbone 
bandwidth, and donations of their own expert time.  They can be 
lifelines for small, independent media and activist organizations who 
can't possibly afford the going commercial rate of > $10k / month for 
ddos protection.

But I would actually much rather see an relatively big organization like 
Avaaz with its own strong fund raising capability raise its own money to 
pay the actual cost for protecting its site than relying on one of these 
subsidized sources (and thus driving out other, smaller potential 
clients of those subsidized sources).  There's obviously need for Avaaz 
to be open about how its raising and spending its money.  But I just 
disagree with the premise that ddos protection is cheap or easy.

-hal

On 5/8/12 1:51 PM, jim youll wrote:
> Having dealt with these problems at various scales (but perhaps not at
> this scale-the facts are fuzzy) i am made very uneasy by the amount of
> money that is claimed both spent and additionally necessary for "DDOS
> protection." Those would be appropriate sums to pay an extortionist as
> "protection money" but they seem to be talking about technology spending
> here, and the whole story is just too much hyperbole and not much that
> seems reasonable at any scale, particularly the overt declaration that
> "DDOS protection" (whatever that means) is a linear function of money
> applied ( above a threshold that imo should have been passed several
> tens of thousands of dollars ago)
>
> Yosem Companys <companys at stanford.edu> wrote:
>
>     *Message from Ricken on Avaaz cyberattack: *
>
>     Hi all - I've heard there's some concern on your list about Avaaz's
>     DDoS trouble. Thanks so much for the offers of help, much
>     appreciated and I know some of you have been great allies in the
>     past, but I think we've got great people working on it and the
>     attack ended last week. Also surprised to hear some of you thought
>     we made this up! If you want to ask a third party, Datagram, Arbor
>     Networks and to lesser degree Croscon were the three groups involved
>     that we asked for advice and help from.
>
>     The other concern I heard is, was this an exaggerated fundraising
>     ploy? Datagram told our tech team it was one of the largest attacks
>     they'd seen, and if we hadn't just 8 weeks ago spent $35k on much
>     fancier DDoS protection it would have completely disabled our site
>     for days. They also said the attacker was constantly adapting to our
>     defenses, the attack was surprisingly sustained, and a key origin
>     appeared to be Amsterdam where we were told some groups for hire
>     operated from - suggesting someone was paying for this. All that
>     triggered our level of concern in writing the fundraiser. Over the
>     last 6 months, we've grown by an average of almost 300,000 people
>     per week, so being disabled for a few days can be super costly. When
>     we brought the guys from Arbor Networks in, they dialed down the
>     concern a little bit, questioning the amsterdam part, and saying it
>     was bigger than the large majority of DDoS attacks, but much larger
>     ones were possible. But that last bit also dialed up our concern,
>     because we knew we were at the limits of what we could handle and we
>     didn't have budget for more. That had been the main reason for the
>     fundraiser.
>
>     And yes, of course we need the money - both for more DDoS protection
>     and also for ramping up our tech security across the board - there
>     was a short list of things in the email. That list also dealt with a
>     wider range of needs, including the physical security of our staff
>     in places like Russia and Lebanon, which also has a tech security
>     component to it. Our community was extremely supportive so we ended
>     up raising more than we need immediately, but this is the first
>     appeal like this we've done in 5 years and we probably won't do
>     another for a long while, so the money has to last. That's part of
>     how online organizing works - you leverage bursts of engagement with
>     particular campaigns and issues to support longer term objectives
>     sustainably. If we find that our plans mean we don't anticipate
>     using a lot of the money for the purpose raised, we email the donors
>     and ask them to either request a refund or tell us what we can use
>     the remainder of t he funds for.
>
>     Hope that helps, and I hope you'll forgive us for a few days delay
>     in replying and not being able to engage and collaborate with you
>     all like we would if we were more a part of your community. We have
>     a small team working in a dozen languages with staff spread across
>     the world, and cover an enormous number of issues in an enormous
>     number of countries. We run about 10-14 campaigns per week, and
>     every campaign we run has a relevant civil society community and
>     often several in different countries (e.g. a French tech community
>     is also demanding our engagement on this one, and even threatening
>     us with a DDoS attack if we don't!). So while I am told that you
>     have norms about collaboration and engagement among you, I regret
>     that we can't follow them. Hope you'll forgive us and judge us by
>     the quality of our work over time. Good luck to you with yours.
>
>     Ricken
>
>
>
> _______________________________________________
> liberationtech mailing list
> liberationtech at lists.stanford.edu
>
> Should you need to change your subscription options, please go to:
>
> https://mailman.stanford.edu/mailman/listinfo/liberationtech
>
> If you would like to receive a daily digest, click "yes" (once you click above) next to "would you like to receive list mail batched in a daily digest?"
>
> You will need the user name and password you receive from the list moderator in monthly reminders. You may ask for a reminder here: https://mailman.stanford.edu/mailman/listinfo/liberationtech
>
> Should you need immediate assistance, please contact the list moderator.
>
> Please don't forget to follow us on http://twitter.com/#!/Liberationtech

-- 
Hal Roberts
Fellow
Berkman Center for Internet & Society
Harvard University