[Bigbang-dev] provenance and sharing collected archives

[Nick Doty]

We've touched on this a couple of times before; I think we've decided not to include collected mailing list archives in the BigBang repository itself. There are few archives that would be relevant to all users, and we're trying to write code for automated collection so that you can download any archive you need for your own research.

That being said, I wonder if it might be useful to have separate repositories where interested researchers can share the archives they've downloaded. I've been downloading mailing list archives for every active W3C Working Group and Interest Group, and separately for every active IETF Working Group; it comes to a lot of data, takes a good deal of time to download and may require some babysitting of those long-running processes. Would others be interested in separate repo's with snapshots of ML archives for those organizations? Or any other common organizations/lists it might be useful to have snapshot data for?

To that point, I also think we'll need useful provenance metadata if we get to the point of sharing archives. When were these downloaded, what was the specific mailing list, what software was used to download them, etc. Indeed, I feel like I should have that functionality just for my individual work in order to maintain good research practice. I opened https://github.com/datactive/bigbang/issues/283 <https://github.com/datactive/bigbang/issues/283> on that 6 weeks ago, and today I've written code to generate provenance.yaml files during the mail collection process: https://github.com/npdoty/bigbang/tree/provenance <https://github.com/npdoty/bigbang/tree/provenance>

I'd appreciate any feedback on the issue or on this list.

I could try to create a minimal PR, but that's getting harder for me as datactive/bigbang's master branch has not been updated in a long time and my code may rely on other changes I've made in intervening months.

Cheers,
Nick
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ghserv.net/pipermail/bigbang-dev/attachments/20170801/18a4d06b/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 496 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://lists.ghserv.net/pipermail/bigbang-dev/attachments/20170801/18a4d06b/attachment.sig>