<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class="">(Niels, I think your email was about this thread, so I'm combining these.)</div><div class=""><br class=""></div><div class="">I think a private GitHub repo, or separate private GitHub repositories for each major organization's archives, would work to start. I'm not sure we could have an unlimited number of private repositories under the datactive GitHub organization without paying an annual fee. We could create the private repositories ourselves and invite collaborators to each individually, or use GitLab, or if datactive has a GitHub account or is willing to pay for one, we can work that way. (I suppose we could also just create a git repository on an accessible server somewhere, but we'd miss the handy user account infrastructure and the like.)</div><div class=""><br class=""></div><div class="">I don't think the examples need to point to an email archive repository; users can run the existing collect_mail scripts to get started on something small. The repository is mostly needed for the really large downloads that encompass many lists. We could mention/point to those repositories in a README somewhere, though.</div><div class=""><br class=""></div><div class="">—Nick</div><div class=""><br class=""></div><div class=""><blockquote type="cite" class=""><div class="">On Aug 24, 2017, at 3:24 AM, Niels ten Oever <<a href="mailto:niels@article19.org" class="">niels@article19.org</a>> wrote:</div><br class="Apple-interchange-newline"><div class="">So, shall we add it to the projects repo or another instance? I can provide server space if that is preferred, but why not go with Github for now. <br class=""><br class="">I think it would make sense to create a different project for that people do not necessarily download all archives when cloning the project.<br class=""><br class="">But would we then point to the Github repo in the example repos?<br class=""><br class="">Best,<br class=""><br class="">Niels<br class=""></div></blockquote></div><br class=""><div><blockquote type="cite" class=""><div class="">On Aug 22, 2017, at 10:33 PM, Harsh Gupta <<a href="mailto:mail@hargup.in" class="">mail@hargup.in</a>> wrote:</div><br class="Apple-interchange-newline"><div class="">
<title class=""></title>
<div class=""><div class="">We can also use gitlab, it has unlimited free private repositories. See <a href="https://about.gitlab.com/gitlab-com/" class="">https://about.gitlab.com/gitlab-com/</a><br class=""></div>
<div class=""><br class=""></div>
<div class=""><div class="signature">Regards<br class=""></div>
<div style="text-align: left;" class="signature">Harsh Gupta<br class=""></div>
</div>
<div class=""><br class=""></div>
<div class=""><br class=""></div>
<div class="">On Wed, 23 Aug 2017, at 03:45 AM, Sebastian Benthall wrote:<br class=""></div>
<blockquote type="cite" class=""><div class=""><div class="">GitHub has private repositories. One could manage permissions through that system.<br class=""></div>
<div class=""><br class=""></div>
<div class="">U.S. IRB says public data isn't human subjects data. I suppose it would fitting if EU was stricter. But I believe even the GDPR says data that's been explicitly made public is fair game.<br class=""></div>
<div class=""><br class=""></div>
<div class="">Another possibility would be versioned cloud storage like an Amazon S3 bucket. There must be a sweet open source equivalent one could set up?<br class=""></div>
</div>
<div class=""><div class=""><br class=""></div>
<div defang_data-gmailquote="yes" class=""><div class="">On Aug 21, 2017 8:43 AM, "Beraldo, Davide" <<a href="mailto:d.beraldo@uva.nl" class="">d.beraldo@uva.nl</a>> wrote:<br class=""></div>
<blockquote defang_data-gmailquote="yes" style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204, 204, 204);padding-left:1ex;" class=""><div class="">Hi guys,<br class=""></div>
<div class=""> <br class=""></div>
<div class=""> first of all, thanks a lot for keeping this on! and apologize for the very long inactivity on this side; resolution for coming academic year is to get more involved with programming for the good (aka not for evil marketing people )<br class=""></div>
<div class=""> <br class=""></div>
<div class=""> on the issue of public repository: i am myself not an ethic fanatic, but working with people who are made me a bit more paranoid; plus, the DATACTIVE project has made some pretty strict ethical commitments with the funders<br class=""></div>
<div class=""> .<br class=""></div>
<div class=""> consequently, i think that making the repositories public would be too much. i nonetheless see the good in having them stored somewhere and let interested people access them.<br class=""></div>
<div class=""> ---would it be possible to have the data stored, listed, but accessible only at request?<br class=""></div>
<div class=""> <br class=""></div>
<div class=""> in the meanwhile i can check with the ethics experts here what they think about it<br class=""></div>
<div class=""> <br class=""></div>
<div class=""> cheers!<br class=""></div>
<div class=""> <br class=""></div>
<div class=""> Davide<br class=""></div>
<div class=""> <br class=""></div>
<div class=""> ______________________________<wbr class="">__________<br class=""></div>
<div class=""> From: Bigbang-dev [<a href="mailto:bigbang-dev-bounces@data-activism.net" class="">bigbang-dev-bounces@data-<wbr class="">activism.net</a>] on behalf of Niels ten Oever [<a href="mailto:niels@article19.org" class="">niels@article19.org</a>]<br class=""></div>
<div class=""> Sent: Sunday, August 20, 2017 2:33 PM<br class=""></div>
<div class=""> To: Nick Doty<br class=""></div>
<div class=""> Cc: <a href="mailto:bigbang-dev@data-activism.net" class="">bigbang-dev@data-activism.net</a><br class=""></div>
<div class=""> Subject: Re: [Bigbang-dev] provenance and sharing collected archives<br class=""></div>
<div class=""> <br class=""></div>
<div class=""> Github sounds good to me, but Davide might have some comments re: (research-)ethics?<br class=""></div>
<div class=""> <br class=""></div>
<div class=""> Cheers,<br class=""></div>
<div class=""> <br class=""></div>
<div class=""> Niels<br class=""></div>
<div class=""> <br class=""></div>
<div class=""> <br class=""></div>
<div class=""> On Fri, Aug 18, 2017 at 03:28:46PM -0700, Nick Doty wrote:<br class=""></div>
<div class=""> > Yeah, separate git repositories sounds like a good way forward. I think having the provenance files will make it easier to collaborate and see the current status of such a data repository.<br class=""></div>
<div class=""> ><br class=""></div>
<div class=""> > Niels, is there a particular reason to use separate server space for these data repositories? Or should we just make them public GitHub repositories? I could potentially see some privacy advantage in not making a public mirror of these mailing list archives -- in the occasional case where public mailing list archive managers remove sensitive messages, our archives wouldn't automatically remove them as well -- but I expect that to be notably rare for these groups that make a point of public archives.<br class=""></div>
<div class=""> ><br class=""></div>
<div class=""> > > On Aug 16, 2017, at 8:31 AM, Sebastian Benthall <<a href="mailto:sbenthall@gmail.com" class="">sbenthall@gmail.com</a>> wrote:<br class=""></div>
<div class=""> > ><br class=""></div>
<div class=""> > > +1 on having data repositories.<br class=""></div>
<div class=""> > > That's a great idea.<br class=""></div>
<div class=""> > ><br class=""></div>
<div class=""> > > Standalone GitHub repositories (not in BigBang but "next to" it) are possible for smaller data sets. Versioning is nice.<br class=""></div>
<div class=""> > ><br class=""></div>
<div class=""> > > Not sure how to do the bigger ones.<br class=""></div>
<div class=""> > ><br class=""></div>
<div class=""> > > On Aug 11, 2017 10:09 AM, "Niels ten Oever" <<a href="mailto:niels@article19.org" class="">niels@article19.org</a> <mailto:<a href="mailto:niels@article19.org" class="">niels@article19.org</a>>> wrote:<br class=""></div>
<div class=""> > > Hi Nick,<br class=""></div>
<div class=""> > ><br class=""></div>
<div class=""> > > I am happy to work on keeping repositories for IETF and ICANN mailinglists. I can also provide server space for the three bodies (W3C, IETF, ICANN), also makes sense because they're connected.<br class=""></div>
<div class=""> > ><br class=""></div>
<div class=""> > > I am very sorry that the Datactive fork is still (far) behind my personal fork. We do want to organize a hackathon on this, RIPE has shown interest in support this work, so hopefully we can organize something to work on this before the end of the year.<br class=""></div>
<div class=""> > ><br class=""></div>
<div class=""> > > Cheers,<br class=""></div>
<div class=""> > ><br class=""></div>
<div class=""> > > Niels<br class=""></div>
<div class=""> > ><br class=""></div>
<div class=""> > ><br class=""></div>
<div class=""> > > On Tue, Aug 01, 2017 at 04:50:03PM -0700, Nick Doty wrote:<br class=""></div>
<div class=""> > > > We've touched on this a couple of times before; I think we've decided not to include collected mailing list archives in the BigBang repository itself. There are few archives that would be relevant to all users, and we're trying to write code for automated collection so that you can download any archive you need for your own research.<br class=""></div>
<div class=""> > > ><br class=""></div>
<div class=""> > > > That being said, I wonder if it might be useful to have separate repositories where interested researchers can share the archives they've downloaded. I've been downloading mailing list archives for every active W3C Working Group and Interest Group, and separately for every active IETF Working Group; it comes to a lot of data, takes a good deal of time to download and may require some babysitting of those long-running processes. Would others be interested in separate repo's with snapshots of ML archives for those organizations? Or any other common organizations/lists it might be useful to have snapshot data for?<br class=""></div>
<div class=""> > > ><br class=""></div>
<div class=""> > > > To that point, I also think we'll need useful provenance metadata if we get to the point of sharing archives. When were these downloaded, what was the specific mailing list, what software was used to download them, etc. Indeed, I feel like I should have that functionality just for my individual work in order to maintain good research practice. I opened <a href="https://github.com/datactive/bigbang/issues/283" class="">https://github.com/datactive/<wbr class="">bigbang/issues/283</a> <<a href="https://github.com/datactive/bigbang/issues/283" class="">https://github.com/datactive/<wbr class="">bigbang/issues/283</a>> <<a href="https://github.com/datactive/bigbang/issues/283" class="">https://github.com/datactive/<wbr class="">bigbang/issues/283</a> <<a href="https://github.com/datactive/bigbang/issues/283" class="">https://github.com/datactive/<wbr class="">bigbang/issues/283</a>>> on that 6 weeks ago, and today I've written code to generate provenance.yaml files during the mail collection process: <a href="https://github.com/npdoty/bigbang/tree/provenance" class="">https://github.com/npdoty/<wbr class="">bigbang/tree/provenance</a> <<a href="https://github.com/npdoty/bigbang/tree/provenance" class="">https://github.com/npdoty/<wbr class="">bigbang/tree/provenance</a>> <<a href="https://github.com/npdoty/bigbang/tree/provenance" class="">https://github.com/npdoty/<wbr class="">bigbang/tree/provenance</a> <<a href="https://github.com/npdoty/bigbang/tree/provenance" class="">https://github.com/npdoty/<wbr class="">bigbang/tree/provenance</a>>><br class=""></div>
<div class=""> > > ><br class=""></div>
<div class=""> > > > I'd appreciate any feedback on the issue or on this list.<br class=""></div>
<div class=""> > > ><br class=""></div>
<div class=""> > > > I could try to create a minimal PR, but that's getting harder for me as datactive/bigbang's master branch has not been updated in a long time and my code may rely on other changes I've made in intervening months.<br class=""></div>
<div class=""> > > ><br class=""></div>
<div class=""> > > > Cheers,<br class=""></div>
<div class=""> > > > Nick<br class=""></div>
<div class=""> > ><br class=""></div>
<div class=""> > ><br class=""></div>
<div class=""> > ><br class=""></div>
<div class=""> > > > ______________________________<wbr class="">_________________<br class=""></div>
<div class=""> > > > Bigbang-dev mailing list<br class=""></div>
<div class=""> > > > <a href="mailto:Bigbang-dev@data-activism.net" class="">Bigbang-dev@data-activism.net</a> <mailto:<a href="mailto:Bigbang-dev@data-activism.net" class="">Bigbang-dev@data-<wbr class="">activism.net</a>><br class=""></div>
<div class=""> > > > <a href="https://lists.ghserv.net/mailman/listinfo/bigbang-dev" class="">https://lists.ghserv.net/<wbr class="">mailman/listinfo/bigbang-dev</a> <<a href="https://lists.ghserv.net/mailman/listinfo/bigbang-dev" class="">https://lists.ghserv.net/<wbr class="">mailman/listinfo/bigbang-dev</a>><br class=""></div>
<div class=""> > ><br class=""></div>
<div class=""> > ><br class=""></div>
<div class=""> > > --<br class=""></div>
<div class=""> > ><br class=""></div>
<div class=""> > > Niels ten Oever<br class=""></div>
<div class=""> > > Head of Digital<br class=""></div>
<div class=""> > ><br class=""></div>
<div class=""> > > Article 19<br class=""></div>
<div class=""> > > <a href="http://www.article19.org/" class="">www.article19.org</a> <<a href="http://www.article19.org/" class="">http://www.article19.org/</a>><br class=""></div>
<div class=""> > ><br class=""></div>
<div class=""> > > PGP fingerprint 2458 0B70 5C4A FD8A 9488<br class=""></div>
<div class=""> > > 643A 0ED8 3F3A 468A C8B3<br class=""></div>
<div class=""> > ><br class=""></div>
<div class=""> > ><br class=""></div>
<div class=""> > > ______________________________<wbr class="">_________________<br class=""></div>
<div class=""> > > Bigbang-dev mailing list<br class=""></div>
<div class=""> > > <a href="mailto:Bigbang-dev@data-activism.net" class="">Bigbang-dev@data-activism.net</a> <mailto:<a href="mailto:Bigbang-dev@data-activism.net" class="">Bigbang-dev@data-<wbr class="">activism.net</a>><br class=""></div>
<div class=""> > > <a href="https://lists.ghserv.net/mailman/listinfo/bigbang-dev" class="">https://lists.ghserv.net/<wbr class="">mailman/listinfo/bigbang-dev</a> <<a href="https://lists.ghserv.net/mailman/listinfo/bigbang-dev" class="">https://lists.ghserv.net/<wbr class="">mailman/listinfo/bigbang-dev</a>><br class=""></div>
<div class=""> > ><br class=""></div>
<div class=""> ><br class=""></div>
<div class=""> <br class=""></div>
<div class=""> <br class=""></div>
<div class=""> <br class=""></div>
<div class=""> --<br class=""></div>
<div class=""> <br class=""></div>
<div class=""> Niels ten Oever<br class=""></div>
<div class=""> Head of Digital<br class=""></div>
<div class=""> <br class=""></div>
<div class=""> Article 19<br class=""></div>
<div class=""> <a href="http://www.article19.org/" class="">www.article19.org</a><br class=""></div>
<div class=""> <br class=""></div>
<div class=""> PGP fingerprint 2458 0B70 5C4A FD8A 9488<br class=""></div>
<div class=""> 643A 0ED8 3F3A 468A C8B3<br class=""></div>
<div class=""> <br class=""></div>
<div class=""> <br class=""></div>
<div class=""> ______________________________<wbr class="">_________________<br class=""></div>
<div class=""> Bigbang-dev mailing list<br class=""></div>
<div class=""> <a href="mailto:Bigbang-dev@data-activism.net" class="">Bigbang-dev@data-activism.net</a><br class=""></div>
<div class=""> <a href="https://lists.ghserv.net/mailman/listinfo/bigbang-dev" class="">https://lists.ghserv.net/<wbr class="">mailman/listinfo/bigbang-dev</a><br class=""></div>
</blockquote></div>
</div>
<div class=""><u class="">_______________________________________________</u><br class=""></div>
<div class="">Bigbang-dev mailing list<br class=""></div>
<div class=""><a href="mailto:Bigbang-dev@data-activism.net" class="">Bigbang-dev@data-activism.net</a><br class=""></div>
<div class=""><a href="https://lists.ghserv.net/mailman/listinfo/bigbang-dev" class="">https://lists.ghserv.net/mailman/listinfo/bigbang-dev</a><br class=""></div>
</blockquote><div class=""><br class=""></div>
</div>
_______________________________________________<br class="">Bigbang-dev mailing list<br class=""><a href="mailto:Bigbang-dev@data-activism.net" class="">Bigbang-dev@data-activism.net</a><br class="">https://lists.ghserv.net/mailman/listinfo/bigbang-dev<br class=""></div></blockquote></div><br class=""></body></html>