IPythonQt end of GSoC status

Enthought approved some funding for our mentors to develop the basis of this big projects that are rising. Yay!
Fernando Perez and Brian Granger (and I think Evan Patterson too) are working on a new communication interface between the kernel and any frontend(kernelmanager or kernel proxy in the diagram), to ease and standarize the work done by every frontend developer.

There're some changes also in the kernel by Fernando and Bryan.
And a reimplementation of the kernel manager by Evan using qt signals instead of callbacks for messages handling.
The floor is a little loose so the building of this scyscraper has to be more careful, and slow written and designed, but it will be for a greater good since its core it's getting robust and trustworthy.
Since the way to communicate with the kernel, and the kernel itself has changed, almost every part of the development before mid-term had to be written again, with a few stones in the way 'cause it depends on a bunch of people (just to mention that Evan and I are sharing QtKernelManager).
A new branch was created http://github.com/muzgash/ipython/tree/ipythonqt-km to implement kernelmanager.

Yesterday I was beginning the tooltip and tab completion support, it's not trivial, and since today is the 'suggested pencils down' date I'm going to improve the documentation and write some tests to fulfill Google and IPython's requirements.

8th week log, Mid-term evaluation!

A lot of details about pdb support and the organization of the output cells were discussed this week.

The characteristics added are:

• The output cells are appended as they appear in the space between th executed cell and the next executable cell. i.e.

for i in range(3):     print i**2

0

1

4

• When you execute again a cell the previous output cells are erased(if any)
• If a new cell is created by the program (not the user) the scroll bar goes to its maximum value. This is not the right way to approach the automatic movement of the viewport I know, I'll work on it on the weekend.
• You can clean the notebook of output cells by clicking a button in evaluate menu.

The last one is part of the details  usually add when my brain is stuck, I try to add a little thing every couple of days.

Today I'm working on subprocess, i.e. communication with the shell. I think I need to create an ipython's instance or inherit InteractiveShell, I'm waiting a hand from Fernando but this shouldn't be too dificult.

I also hope to finish everything planned round next week to put some effort on the highlighting and autoindentation.

Behind raw_input

This was a pretty heavy week.

there were two ways of approaching the raw_input problem, the right one, and the wrong one(lol):

• To fill the widget definition with signals and slots which will communicate the different IOs to the mainwindow and this one then will communicate those IOs to the kernel.
• To import zmq in the ipqt_popup module and manage the communication directly with the kernel.

I innocently chose the fist one (FAIL), "entia non sunt multiplicanda praeter necessitatem".

The widget was almost done when Fernando corrected me the design stating that it will be better to print all the messages from raw_input or pdb in the mainwidow not in a popped up widget.

Fortunately I managed to understand the mechanisms required for this to work correctly and I'm finishing it right now.

Some screenshots

Bryan suggested some screen shots of what's working. Just a pair of very, very simple examples:

Here we can see the main function of the frontend (run code) and some of its code in the back. (Yes, I'm very used to graphical editors)

And here we can see my yakuake terminal with some useless stuff and a terminal frontend for ipython which connects to the same kernel (i.e.communicate through the same ports), this one changing the value of 'a' which has been already assigned in the IPythonQt window, so a message appears in the new widget on the right (plz don't heed the preview).

Added features to June 29

What can you do?
When you open IPythonQt you see a menubar and a (still undeletable) input cell in which you can write ipython/python code.

You can execute the code written by pressing Ctrl+Return (shift doesn't work, I dunno why), or just go to the menu 'evaluate'.

You can call a context menu inside the cell which besides the standard buttons it has a new one to delete the cell.

If you right-click somewhere else in the notebook, a tiny context menu should appear to append a new input cell or to delete the last focused cell. A focused cell is the last you clicked, not the last you hover.

If you connect another frontend to the same kernel, another widget will come out with the info of the variables that changed and the user.

Some shortcuts has been created for the keyboard lover:

Ctrl+UpArrow goes to the previous cell

Ctrl+DownArrow goes to the next cell

Ctrl+Shift+UpArrow goes to the previous executable cell

Ctrl+Shift+DownArrow goes tot he next executable cell

Ctrl+Shift+N appends a new cell

Ctrl+Shift+Delete removes the current cell

Ctrl+Return Executes the current cell if it's executable

Ctrl+Q Quits

What you cannot do (yet):

The code still doesn't capture pyout, so code like:

_cell__

|a=10  |

|a        |

won't produce any output.

Calls to raw_input, %, !, system commands, tab completion, function tooltips, gdb and help.

Work right in progress:

Stdin support, since I have to have it finished for the mid-term evaluation, it's the main concern. The popup widget it's finished but there's still the raw_input function in the kernel that needs to be implemented correctly.

There's a new output cell if the same input cell is modified.

There's a discussion in the mailing list about if the different outputs from the same input should be on different cells (packable) or in the same cell.

The resizing of the cell when the block(line without '/n') is larger than the size of the window. For what I've seen this is not trivial.

Testing IPythonQt

For you to test IPythonQt you first need to set up some stuff first:

1. Install the following: git g++ cython uuid-dev python-qt4-dev pyqt4-dev-tools , I think that would be enough.
2. You also need to install zeromq and pyzmq from sources here's a good guide to do that.
3. Take into account that you need python2.6.

Now you can download IPythonQt's code from http://github.com/muzgash/ipython/tree/ipythonqt
check if you are in the right_branch (ipythonqt, not master): git branch
if not right_branch:
    git checkout ipythonqt

cd ipython/IPython/gui/qt

run ./kernel.py

Open a new terminal and run ./ipythonqt.py

Obviously it's not working with all its features, I just remove some dock widgets that were in the original design of the main view, but they're just commented because they don't do anything yet.

Repositories GitHub

You can check the progress at GitHub:
http://github.com/muzgash/ipython

The commits are a little slow because some issues with GitHub and its ssh keys

Possible future directions

The near future will bring the feature of saving and loading sessions, also importing and exporting to different formats like rst, html, pdf and python/ipython code, a discussion about this is taking place in the ipython-dev mailing list. Also the interaction with a remote kernel and distributed computation which is an IPython's project already in development.

The idea of a mathematica-like help widget (i.e. there will be parts of it that will execute as a native session of IPythonQt) is still to be discussed in the development mailing list but it's definitively a great idea.

Preliminary work

I have already developed a prototype using Qt that provides individual QTextEdit widgets for each block of user input. This makes it possible to have an editable cell be independent from the area where output or errors are displayed (which can't be achieved if a single widget like QTableWidget is used for the entire UI). The screenshot above shows this basic interface, where formatting and write control are handled independently for each area.
I also explored other tools such as QScintilla and OpenModelica (wrapping its interface with Cython) but due to license incompatibilities (GPL) decided against using these and will rely solely on Qt.

Schedule

• Week 1: Simple client/server pyzmq module implemented from the previous work to the GUI needs.
• Week 2 and 3: Multiline input in a cell form that is sent to the kernel as a string, each cell has its own widget so they can be edited indepently.
• Week 4-5: Capture all output streams from kernel and properly format exceptions and errors. Display streams corresponding to other clients that may be connected to the same kernel in a separate widget so they don't confuse the user. This output will be asynchronous, so it should be handled in the GUI without blocking the user input widget.
• Week 6: Stdin support: only one frontend at a time can communicate with raw_input calls made by the kernel. API for frontends to grab stdin.
• Week 7: debugger support. IPython calls a modified pdb, which reads from stdin using raw_input. So once raw_input handling is ready, it becomes possible to redirect the pdb session from the kernel to the GUI.
• Week 8: Subprocess support (print output of 'ls' in gui, for example).
• Week 9: Tab completion and function tooltips (GUI popup with the function details). Must gracefully timeout and avoid trying if kernel is busy
• Week 10: padding for any slip in previous schedule.

More details

2-PROCESS MODEL PYZMQ BASE
Since the necessity of a user to keep his data safe, the design is based in a 2-process model that will be achieved with a simple client/server system with pyzmq, so the GUI session do not crash if the the kernel process does. This will be achieved using this test code and customizing it to the necessities of the GUI such as queue management with discrimination for different frontends connected to the same kernel and tab completion. A piece of drafted code for the kernel (server) should look like this:

def main():
    c = zmq.Context(1, 1)
    ip = '127.0.0.1'
    port_base = 5555
    connection = ('tcp://%s' % ip) + ':%i'
    rep_conn = connection % port_base
    pub_conn = connection % (port_base+1)
    print >>sys.__stdout__, "Starting the kernel..."
    print >>sys.__stdout__, "On:",rep_conn, pub_conn
    session = Session(username=u'kernel')
    reply_socket = c.socket(zmq.XREP)
    reply_socket.bind(rep_conn)
    pub_socket = c.socket(zmq.PUB)
    pub_socket.bind(pub_conn)
    stdout = OutStream(session, pub_socket, u'stdout')
    stderr = OutStream(session, pub_socket, u'stderr')
    sys.stdout = stdout
    sys.stderr = stderr
    display_hook = DisplayHook(session, pub_socket)
    sys.displayhook = display_hook
    kernel = Kernel(session, reply_socket, pub_socket)


This kernel will use two queues (output and input), the input queue will have id of the process(frontend) making the request, type(execute, complete, help, etc) and id of the request itself and the string of code to be executed, the output queue will have basically the same information just that the string is the to be displayed. This model is because the kernel needs to maintain control of timeouts when multiple requests are sent and keep them indexed.

QT BASED GUI
Design of the interface is going to be based in cells of code executed on the previous defined kernel. It will also have GUI facilities such toolboxes, tooltips to autocomplete code and function summary, highlighting and autoindentation. It will have the cell kind of multiline edition mode so each block of code can be edited and executed independently, this can be achieved queuing QTextEdit objects (the cell) giving them format so we can discriminate outputs from inputs. One of the main characteristics will be the debug support that will show the requested outputs as the debugger (that will be on a popup widget) "walks" through the code, this design is to be reviewed with the mentor.


The GUI will check continuously the output queue from the kernel for new information to handle. This information have to be handled with care since any output will come at anytime and possibly in a different order than requested or maybe not appear at all, this could be possible due to a variety of reasons(for example tab completion request while the kernel is busy processing another frontend's request). This is, if the kernel is busy it won't be possible to fulfill the request for a while so the GUI will be prepared to abandon waiting for the reply if the user moves on or a certain timeout expires.

Main window proposal

This is a tentative view of the main window.

I hope to see your comments on this.

IPython Qt interface - GSoC 2010 Proposal

ABSTRACT

I will implement a Qt-based Graphical User Interface (GUI) to execute Python code with an interpreter that runs in a separate process and the two systems (GUI frontend and interpreter kernel) communicating via the ZeroMQ Messaging library. The bulk of the implementation will be done without dependencies on IPython (only on Zmq). Once the key features are ready, IPython-specific features can be added using the IPython codebase.

PROJECT DETAILS

For a long time there has been demand for a graphical user interface for IPython, and the project already ships Wx-based prototypes thereof. But these run all code in a single process, making them extremely brittle, as a crash of the Python interpreter kills the entire user session. Here I propose to build a Qt-based GUI that will communicate with a separate process for the code execution, so that if the interpreter kernel dies, the frontend can continue to function after restarting a new kernel (and offering the user the option to re-execute all inputs, which the frontend can know).

This GUI will allow for the easy editing of multi-line input and the convenient re-editing of previous blocks of input, which can be displayed in a 2-d workspace instead of a line-driven one like today's IPython. This makes it much easier to incrementally build and tune a code, by combining the rapid feedback cycle of IPython with the ability to edit multiline code with good graphical support.

More info: http://ipython.scipy.org/moin/GSoC2010/IPythonQt (it's sometimes down)