[liberationtech] University Oral Exam: Michael Fischer - TOMORROW. Thursday, June 11th, Noon PST

[Michael Fischer]

*End-User Programming of Virtual Assistants Skills with Stylish Graphical
User Interfaces*
Michael Fischer
Advised by Professor Monica Lam
Computer Science Department

Oral Exam
Public Session: Noon – 1:00PM PST
Thursday, June 11th 2020
Location:
https://stanford.zoom.us/j/95233558336?pwd=eExOa1FOMWVqWXkxNm9Ud0hSOFhPUT09

*Summary*
Virtual assistants give end-users the capability to access their devices
and web data using hundreds of thousands of predefined skills. Nonetheless,
there is still a long-tail of personal digital tasks that individuals wish
to automate.  This thesis explores how end-users can define useful
personalized skills with designer-like interfaces, all without learning any
formal programming languages.

Our system enables end-users to develop virtual assistant skills in their
web-browser by capturing what they say, type, and click on.  This system is
the first program-by-demonstration system that produces programs with
control constructs.  The system gives the user an easy-to-learn multimodal
interface and generates code in a formal programming language which
supports parameterization, function invocation, conditionals, and iterative
execution.

We show that a virtual assistant skill can greatly benefit from having a
graphical interface as users can monitor multiple queries simultaneously,
re-run skills easily, and adjust settings using multiple modes of
interaction.  We developed a system that automatically translates a user’s
voice command into a reusable skill with a graphical user interface.
Unlike the formulaic interfaces generated by prior state of the art work,
we generate interfaces that are visually interesting and diverse by using a
novel template-based approach.

To improve the aesthetics of graphical user interfaces we use a technique
called style transfer, a method for applying the style of one image to
another.  We show that the previous formulation of style transfer cannot
retain structure in an image, which causes the output result to lack
definition and legibility and renders restyled interfaces not usable. Our
purely neural-network based solution captures structure by the uncentered
cross-covariance between features across different layers of a
convolutional neural network. By minimizing the loss between the style and
output images, our technology retains structure while generating results
with texture in the background, shadow and contrast at the borders, and
consistency of design across edges.

In summary, our system enables end-users to create web-based skills with
designer-like automatically generated graphical user interfaces.

[image: ad.png]
image from United States vs. Alexa
<https://mrs.stanford.edu/art-science-2020-exhibition#videos> project
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ghserv.net/pipermail/lt/attachments/20200610/cb858b27/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ad.png
Type: image/png
Size: 2944326 bytes
Desc: not available
URL: <http://lists.ghserv.net/pipermail/lt/attachments/20200610/cb858b27/attachment.png>