Any comments on setting this up for testing etc. @jasongrout?

It doesn't work just yet (I was able to get an image to display, but not the interactive tools to load). You'll need IPython with ipython/ipython#4195 applied. And actually, there were some changes in the IPython pull request that may necessitate changes in this pull request. Also, I was testing against the Sage Cell Server---the IPython notebook will be very similar, but I didn't have time to test against it yet.

Probably the first thing to do would be to set up IPython with that pull request mentioned above and get familiar with the Comm API (there are some simple examples on the ticket). Then updating this to use the refactored communication framework that Michael pointed out on the mailing list.

I updated this and it works now with IPython master. Execute this in an IPython notebook cell

from matplotlib.backends.backend_webagg_core import (
    FigureManagerWebAgg, new_figure_manager_given_figure)
import json

from IPython.display import display,Javascript,HTML
from IPython.kernel.comm import Comm
from uuid import uuid4 as uuid
from base64 import b64encode

display(Javascript(FigureManagerWebAgg.get_javascript()))

class CommFigure(object):
    def __init__(self, figure):
        self.figure = figure
        self.manager = new_figure_manager_given_figure(id(figure), figure)
        self.comm = CommSocket(self.manager)
        self.comm.open()


class CommSocket(object):
    """
    A websocket for interactive communication between the plot in
    the browser and the server.

    In addition to the methods required by tornado, it is required to
    have two callback methods:

        - ``send_json(json_content)`` is called by matplotlib when
          it needs to send json to the browser.  `json_content` is
          a JSON tree (Python dictionary), and it is the responsibility
          of this implementation to encode it as a string to send over
          the socket.

        - ``send_binary(blob)`` is called to send binary image data
          to the browser.
    """
    supports_binary = False

    def __init__(self, manager):
        self.manager = manager
        self.uuid = str(uuid())
        display(HTML("<div id='%s'></div>"%self.uuid))
        self.comm = Comm('matplotlib', data={'id': self.uuid})

    def open(self):
        # Register the websocket with the FigureManager.
        self.manager.add_web_socket(self)
        self.comm.on_msg(self.on_message)

    def on_close(self):
        # When the socket is closed, deregister the websocket with
        # the FigureManager.

        self.manager.remove_web_socket(self)
        self.comm.close()

    def send_json(self, content):
        self.comm.send({'data': json.dumps(content)})

    def send_binary(self, blob):
        data_uri = "data:image/png;base64,{0}".format(b64encode(blob))
        self.comm.send({'data': data_uri})

    def on_message(self, message):
        # The 'supports_binary' message is relevant to the
        # websocket itself.  The other messages get passed along
        # to matplotlib as-is.

        # Every message has a "type" and a "figure_id".
        message = json.loads(message['content']['data'])
        if message['type'] == 'supports_binary':
            self.supports_binary = message['value']
        else:
            self.manager.handle_json(message)


Javascript("""
var comm_websocket = function(comm) {
    var ws = {};
    // MPL assumes we have a websocket that is not open yet
    // so we run the onopen handler after they have a chance
    // to set it.
    ws.onopen = function() {};
    setTimeout(ws.onopen(), 0);
    ws.close = function() {comm.close()};
    ws.send = function(m) {
        comm.send(m); 
        console.log('sending',m);
    };
    comm.on_msg(function(msg) {
        console.log('receiving', msg);
        ws.onmessage(msg['content']['data'])
    });
    return ws;
}
var figures = [];
mpl.mpl_figure_comm = function(comm, msg) {
var id = msg.content.data.id;
var element = $("#"+id);
var c = comm_websocket(comm)
    var m = new mpl.figure(id, c,
       function() {console.log('download')}, element.get(0));
    figures.push(m);
    container.show();
}
IPython.notebook.kernel.comm_manager.register_target('matplotlib', mpl.mpl_figure_comm);
""")

and then you can do something like:

from matplotlib.figure import Figure
import numpy as np
fig = Figure()
a = fig.add_subplot(111)
t = np.arange(0.0, 3.0, 0.01)
s = np.sin(2 * np.pi * t)
a.plot(t, s)
CommFigure(fig)

Here are a few points to still be cleaned up:

• on the javascript side, we prevent garbage collection by adding everything to a figures list. This probably doesn't work, since the figures list is a local variable. Maybe we ought to add them to the global mpl object
• Since matplotlib assumes the socket just transmits strings, it is stringifying things, which we then stringify again to put into IPython's message dict. It would be better if matplotlib could use a 'smarter' socket that could transmit in json.

This is looking really good. Obviously there are a couple of trivial UI enhancements to add (the button icons, resize, download button doesn't work etc.) and it 'feels' like there is a latency issue which I didn't have with the WebAgg backend (perhaps that is the cost of using base 64 vs binary) but generally this is and absolutely huge step towards the feature that we all want to see in matplotlib/IPython. I think the next steps are to look at some of the issues above, and to thrash out whether we need to consider compressing the data payload (and decompressing in JS on the other side). It'd also be nice if server side rendering errors (which I got because I didn't build mpl properly after updating) were tracebacked on the client, and another nice to have would be to have some sort of debug panel, which can tell us the FPS etc.

Awesome stuff. What are your plans from here @jasongrout?

On 11/15/13 4:01 AM, Phil Elson wrote:

This is looking really good. Obviously there are a couple of trivial UI
enhancements to add (the button icons, resize, download button doesn't
work etc.) and it 'feels' like there is a latency issue which I didn't
have with the WebAgg backend (perhaps that is the cost of using base 64
vs binary) but generally this is and absolutely huge step towards the
feature that we all want to see in matplotlib/IPython. I think the next
steps are to look at some of the issues above, and to thrash out whether
we need to consider compressing the data payload (and decompressing in
JS on the other side).

The image data is already compressed (png). I agree that it still feels
a bit laggy (though I haven't compared it to the WebAgg backend yet).

It'd also be nice if server side rendering errors
(which I got because I didn't build mpl properly after updating) were
tracebacked on the client, and another nice to have would be to have
some sort of debug panel, which can tell us the FPS etc.

+1

Awesome stuff. What are your plans from here @jasongrout
https://github.com/jasongrout?

Get it working in the Sage cell server.

...

Done!

http://sagecell.sagemath.org/?q=ilpajg (for a sage plotting example)

http://sagecell.sagemath.org/?q=spadja (for one of the examples from the
matplotlib docs)

@jasongrout - is it worth merging the machinery that you've had to add even if we don't actually have the CommFigure class?

I think probably not. Let's wait to get a commfigure class. I've been working a ton with the IPython folks on their widget infrastructure, and it's helping me understand how best to approach things.

Closing this as I this was the base of nbagg which has sense been merged.