[Bigbang-dev] provenance and sharing collected archives

[Nick Doty]

Hi all,

I've opened a pull request with a provenance metadata file creation feature; it also has a couple error-handling and documentation improvements.
https://github.com/datactive/bigbang/pull/289 <https://github.com/datactive/bigbang/pull/289>

I hope this can be useful to all and that we can get it reviewed and integrated promptly.

(I believe I was able to use git rebase on top of master so that it only includes the relevant commits; it makes for easier reviewing and doesn't require cherry-picking or messy additional commits to remove other scientific files that I was also working on. git rebase is a powerful but scary tool, so while I recommend this, I recommend doing it with caution and not on any branches/commits that have already been pushed to the main datactive repo.)

Regarding shared archives, I'm finding that the number and size of the files is making straight-up checking them into Git a little difficult; it takes minutes just to git add the files to a changelist. I suspect that the git-lfs extension would be a useful way to help with this. Under my understanding, we would, in short, check in hashes of files to a git repository and then the full additional mail archives to another location (hosted by GitHub or GitLab) and then git-lfs will download the full mail archives (but not every version of them) as needed. If that sounds reasonable, then I think we can more easily pursue the shared archives approach (for all the IETF or ICANN archives, say) with non-public hosting, either on GitLab or through a separate server as Niels had offered. I'll investigate more and let people know; if anyone on the list has experience with git-lfs (or, alternatively, git-annex) already, please let me know!

Cheers,
Nick

> On Aug 1, 2017, at 4:50 PM, Nick Doty <npdoty at ischool.berkeley.edu> wrote:
> 
> We've touched on this a couple of times before; I think we've decided not to include collected mailing list archives in the BigBang repository itself. There are few archives that would be relevant to all users, and we're trying to write code for automated collection so that you can download any archive you need for your own research.
> 
> That being said, I wonder if it might be useful to have separate repositories where interested researchers can share the archives they've downloaded. I've been downloading mailing list archives for every active W3C Working Group and Interest Group, and separately for every active IETF Working Group; it comes to a lot of data, takes a good deal of time to download and may require some babysitting of those long-running processes. Would others be interested in separate repo's with snapshots of ML archives for those organizations? Or any other common organizations/lists it might be useful to have snapshot data for?
> 
> To that point, I also think we'll need useful provenance metadata if we get to the point of sharing archives. When were these downloaded, what was the specific mailing list, what software was used to download them, etc. Indeed, I feel like I should have that functionality just for my individual work in order to maintain good research practice. I opened https://github.com/datactive/bigbang/issues/283 <https://github.com/datactive/bigbang/issues/283> on that 6 weeks ago, and today I've written code to generate provenance.yaml files during the mail collection process: https://github.com/npdoty/bigbang/tree/provenance <https://github.com/npdoty/bigbang/tree/provenance>
> 
> I'd appreciate any feedback on the issue or on this list.
> 
> I could try to create a minimal PR, but that's getting harder for me as datactive/bigbang's master branch has not been updated in a long time and my code may rely on other changes I've made in intervening months.
> 
> Cheers,
> Nick

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ghserv.net/pipermail/bigbang-dev/attachments/20171025/558a127e/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 529 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://lists.ghserv.net/pipermail/bigbang-dev/attachments/20171025/558a127e/attachment.sig>