[liberationtech] Censorship hardware - BLUECOAT IN SYIA

[Maxim Kammerer]

On Tue, Dec 4, 2012 at 7:38 PM,  <liberationtech at lewman.us> wrote:
> A few people started to dig through the data, and then either gave up
> when they realized the volume of it, or didn't publish their analysis
> widely.  Here's one example,
> http://picviz.blogspot.com/2012/01/syrian-bluecoat-logs-analysis-part-1.html
>
> Blue Coat logs are just ELFF format, nearly anything can parse them and
> make pretty reports good enough for enterprise bosses. The value comes
> from understanding what's missing in the logs, what's being tracked
> overall, and who is communicating with whom. 500GB isn't that much
> data. One could just take the raw logs, parse and import them into a
> SQL database and then generate queries until the cows come home.

I doubt you will find anything useful, besides maybe
reverse-engineering the rules for forwarding requests to the Blue Coat
devices. Only the first six short SG-42 files [1] contain requests
with hashed user IPs (436 MiB, 6.4M entries), and the rest have
c-ip=0.0.0.0, apparently generated by some anti-virus software [2].
Think you can use User-Agent to distinguish between the boring users
who have parental control software? Good luck: for an (arbitrary) file
with 25M requests, there are just 65K distinct User-Agent strings
(looks like enough, but distribution will be far from uniform). So you
can find out, from the short SG-42 files, that user 30a5f2f9049b9981
watched a really impressive amount of porn in one day. Amazing! And
boring. About the only useful thing that can be done with the dataset
is reverse engineering the rules for filtering and blocking URLs.

[1] http://project-bluesmote.s3-website-us-east-1.amazonaws.com/raw_logs/SG-42/SG_main__4207{22212535,22222039,22231541,23002434,23084209,23153411}.log.gz
[2] http://reflets.info/bluecoats-presence-in-syria-finally-uncovered/

-- 
Maxim Kammerer
Liberté Linux: http://dee.su/liberte