[liberationtech] [drone-list] US DoD Directive on Autonomy

[Eugen Leitl]

----- Forwarded message from Gregory Foster <gfoster at entersection.org> -----

From: Gregory Foster <gfoster at entersection.org>
Date: Mon, 26 Nov 2012 09:26:21 -0600
To: drone-list at lists.stanford.edu
Subject: [drone-list] US DoD Directive on Autonomy
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7;
	rv:17.0) Gecko/17.0 Thunderbird/17.0

Secrecy News (Nov 26) - "Autonomy in Weapon Systems":
http://www.fas.org/blog/secrecy/2012/11/weapons_autonomy.html

One note.  Mention is made in this document about the importance of  
"verification and validation" (V&V) of systems.  Earlier this month,  
DARPA's High-Assurance Cyber Military Systems (HACMS) program awarded a  
4.5 year $18M contract to an international consortium "to develop a  
complete, formally proven architecture to protect the control and  
communication systems of an aerial vehicle from compromise by faults and  
targeted attacks."
http://www.nicta.com.au/media/current/nicta_to_develop_critical_software_for_multi-million-dollar_us_government_cyber-security_project

Part of that award went to National ICT Australia [ @NICTA ] which toots  
its own horn for formally verifying 7,500 lines of C code. That tells you 
something about the challenge of "formal verification" which  
mathematically proves a system will behave as specified - a much higher  
bar than "rigorous" testing.
http://en.wikipedia.org/wiki/Formal_verification

That also tells you something about the complexity of systems which have  
thus far achieved formal verification.  For comparison, the Linux  
kernel---which sits at the heart of many of the workstations used by the  
US military's remote pilots---surpassed 15 million lines of code in  
January, leading the creator of Linux, Linus Torvalds, to lament that he  
is "afraid of the day" when there will be an error that "cannot be  
evaluated anymore."
http://www.tomshardware.com/news/Linux-Linus-Torvalds-kernel-too-complex-code,14495.html

Considering DARPA knows the path to formal verification of larger systems 
must be cleared - demand it.  The challenge of formal verification presents 
a strong argument for delaying deployment of lethal autonomous systems.

gf

-- 
Gregory Foster || gfoster at entersection.org
@gregoryfoster <> http://entersection.com/

_______________________________________________
drone-list mailing list
drone-list at lists.stanford.edu

Should you need to change your subscription options, please go to:

https://mailman.stanford.edu/mailman/listinfo/drone-list

If you would like to receive a daily digest, click "yes" (once you click above) next to "would you like to receive list mail batched in a daily digest?"

You will need the user name and password you receive from the list moderator in monthly reminders.

Should you need immediate assistance, please contact the list moderator.

----- End forwarded message -----
-- 
Eugen* Leitl <a href="http://leitl.org">leitl</a> http://leitl.org
______________________________________________________________
ICBM: 48.07100, 11.36820 http://www.ativel.com http://postbiota.org
8B29F6BE: 099D 78BA 2FD3 B014 B08A  7779 75B0 2443 8B29 F6BE