<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Hi Niels,<br class=""><div><br class=""><blockquote type="cite" class=""><div class="">On Mar 19, 2020, at 6:58 AM, Niels ten Oever <<a href="mailto:mail@nielstenoever.net" class="">mail@nielstenoever.net</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><span style="caret-color: rgb(0, 0, 0); font-family: LucidaGrande; font-size: 11px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">- The focus of my work recently has shifted from ICANN, IETF and RIPE to 3GPP and the IEEE. Unfortunately these organizations don't use Mailman, but L-Soft's Listserv 16.5 (</span><a href="https://list.etsi.org/scripts/wa.exe" style="font-family: LucidaGrande; font-size: 11px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">https://list.etsi.org/scripts/wa.exe</a><span style="caret-color: rgb(0, 0, 0); font-family: LucidaGrande; font-size: 11px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class=""><span class="Apple-converted-space"> </span>and<span class="Apple-converted-space"> </span></span><a href="https://listserv.ieee.org/cgi-bin/wa?HOME" style="font-family: LucidaGrande; font-size: 11px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">https://listserv.ieee.org/cgi-bin/wa?HOME</a><span style="caret-color: rgb(0, 0, 0); font-family: LucidaGrande; font-size: 11px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">). Is there a way we could scrape these archives as well?</span><br style="caret-color: rgb(0, 0, 0); font-family: LucidaGrande; font-size: 11px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""></div></blockquote></div><br class=""><div class="">L-Soft Listserv is a new one for me, I hadn’t seen mailing list archives like this before.</div><div class=""><br class=""></div><div class="">If you can somehow download .mbox files or plain text archives from these groups, that would clearly be easiest. If not, though, I think you could write a small scraper for their online format and then it would be fairly easy to integrate into bigbang.</div><div class=""><br class=""></div><div class="">For W3C archives, I have w3crawl.py, which follows the links to individual message pages in W3C’s online archives, which use a version of pipermail that is pretty specific to them. We don’t have this formally as a subclass in BigBang, but we can and should do that at some point. For now, mailman.py just tries to determine from the URL which crawler it should use, and switches to w3crawl.py explicitly when it looks to be a match.</div><div class=""><br class=""></div><div class="">You’d need to make a subclass of email.parser.Parser to create a single email message from a string, and then a version of collect_from_url() to handle the steps of finding all the messages for a list and parsing each. Then bigbang will save it as an mbox and it’s easy to parse whenever.</div><div class=""><br class=""></div><div class="">Hope this helps,</div><div class="">Nick</div></body></html>