FULL PAPER Magnetic Resonance in Medicine 74:1449–1460 (2015)

Graphical Programming Interface: A Development Environment for MRI Methods Nicholas R. Zwart* and James G. Pipe Purpose: To introduce a multiplatform, Python languagebased, development environment called graphical programming interface for prototyping MRI techniques. Methods: The interface allows developers to interact with their scientific algorithm prototypes visually in an event-driven environment making tasks such as parameterization, algorithm testing, data manipulation, and visualization an integrated part of the work-flow. Algorithm developers extend the built-in functionality through simple code interfaces designed to facilitate rapid implementation. Results: This article shows several examples of algorithms developed in graphical programming interface including the non-Cartesian MR reconstruction algorithms for PROPELLER and spiral as well as spin simulation and trajectory visualization of a FLORET example. Conclusion: The graphical programming interface framework is shown to be a versatile prototyping environment for developing numeric algorithms used in the latest MR techniques. C 2014 Wiley PeriodMagn Reson Med 74:1449–1460, 2015. V icals, Inc. Key words: graphical programming; reconstruction; spin simulation

INTRODUCTION Software development and deployment on magnetic resonance imaging systems can be grouped into two categories. Specific software is used to drive the scanner, and often a separate piece of software is used to reconstruct the sampled data into images. The work presented in this article primarily focuses on the latter, specifically on the development environment used to generate algorithms for MR techniques. The purpose of MR image reconstruction software is to efficiently construct images from acquired data using numeric algorithms. The efficiency of the software is determined by computational duration and hardware requirements. New reconstruction software developed within the scientific community (1–6) and new vendor software releases have improved on these efficiency metrics by also focusing on the development aspect of the Keller Center for Imaging Innovation, Barrow Neurological Institute, Phoenix, Arizona, USA. Grant sponsor: Philips Healthcare. *Correspondence to: Nicholas R. Zwart, Ph.D., Keller Center for Imaging Innovation, Barrow Neurological Institute, Phoenix, AZ, 85013. E-mail: [email protected] Received 26 June 2014; revised 17 October 2014; accepted 20 October 2014 DOI 10.1002/mrm.25528 Published online 10 November 2014 in Wiley Online Library (wileyonlinelibrary.com). C 2014 Wiley Periodicals, Inc. V

software itself. These projects have accomplished this through both accessibility and modularity. In MR research, reconstruction algorithm development is driven by new applications and new acquisitions, requiring continual updates to the reconstruction. Modularity provides developers with the ability to quickly implement new methods by providing a framework for reusing elements of existing methods without difficulty. Algorithm elements developed in this fashion are more easily reused if they are generalized and benefit all dependent algorithms as they are updated. Modular frameworks also implicitly enable distributed development within a team or community. Reconstruction frameworks that encompass these qualities are effectively algorithm development environments. The framework provides a basic structure that enables code to be organized, assembled, and bound together in a specific composition. In a more abstract way, all algorithms that fit the base structure can be developed on the given framework. An algorithm development framework can also facilitate prototyping by providing tools for rapid integration, algorithm exploration, and data visualization. These elements are the focus of the software presented in this article, which is an algorithm development environment called the graphical programming interface (GPI) (7,8). GPI provides a framework for assembling and analyzing algorithms in a visual interactive format. MR algorithms such as image reconstruction, spin simulation, gradient, and RF design are well represented as flow diagrams, or more specifically, directed acyclic graphs (9). In GPI, these graphs can be constructed and manipulated interactively to investigate, perform tests on individual elements, or visualize data at various points within the graph. This work shows how the graphical nature is suited for algorithm development as well as teaching. METHODS GPI is a multiplatform, Python (10)-based, application framework similar to projects such as Advanced Visual Systems (AVS) and Vision (9,11). The framework provides a mechanism for algorithm development both graphically, using the built-in library, and programmatically using the Python and Cþþ languages. The graphical and code-based interfaces described in the following sections are designed to minimize the developer side coding effort and provide tools to help expedite algorithm development. Graphical Interface The graphical interface is written in Python using the PyQt (12) bindings for the Qt application framework. Figure 1 shows a screenshot of the GPI “canvas” with some

1449

1450

Zwart and Pipe

FIG. 1. A screenshot of an active GPI session with UI element labels.

of the user interface (UI) elements-labeled. Algorithm elements, called “nodes,” are instantiated on the canvas and connected (via mouse input) to create a processing “network” as represented by a flow diagram. Nodes can be instantiated on the canvas through a node-library context menu or by dragging and dropping the node code file from a file browser onto the canvas. As the underlying node-code is being developed, the node can be repeatedly re-instantiated to check the development progress without restarting GPI. Figure 2a shows some of the tools available in the canvas. Tool tips for each node show the process execution time (wall time), average execution time, and the amount of system memory held by the node.

Each node contains “port” elements (shown at the top and bottom of each node in Fig. 1), which represent the input and output of the numeric algorithm implemented within. Nodes are connected to each other through their specific ports. Ports can specify restrictions on data parameters such as type, dimensionality, size, and so forth depending on the requirements of algorithm. These port parameters are enforced by disallowing the connections between ports with differing restrictions. This port specification abstracts the algorithm element from the application, allowing users to potentially use the node for algorithms outside of the developer’s original intended use, enabling more sophisticated algorithm

FIG. 2. Screenshots of various canvas informational and runtime feedback displays. a: Hovering over a node, port or edge will provide tooltips with type, timing and memory usage about the node. b–c: Port colors indicate whether a port has new data (blue), no data (red) or unchanged data (yellow). d: A dark gray node indicates that the node is currently executing. e: The warning code (yellow) discontinues the network execution and waits for a retry. f: The error code (red) pauses the canvas (yellow) and will not attempt to rerun the node.

Graphical Programming Interface

exploration (i.e. if the port connections are satisfied, the node will run). Figure 2a–c shows the port and edge tool tips. The port tip provides the port label (as in the code) and data type information while the edge provides a shorthand description for each data type (such as dimensionality for array data), enabling quick identification of compatible ports during node assembly. The ports also have a color code that indicates new data (blue), unchanged data (yellow), and no data (red), providing the user with the I/O status of the underlying algorithm. One of the primary features of the UI is the ability to parameterize the node algorithm code using an interface that is simple to build and use. Each node has an associated interactive menu that allows algorithm parameters to be modified (through sliders, text boxes, push buttons, etc.). These UI elements are called “widgets.” Widgets can also be used for displaying information such as images, plots, or text. Like ports, widgets provide a fixed mechanism for parameterizing a node’s algorithm. The widget can be defined with limits allowing the developer to make the user interaction robust and prevent misparameterization. This can minimize the time needed by the user to learn how to integrate a specific algorithm element into their own work. Changes, to the canvas or node menu, trigger new processing events that update each algorithm in realtime. Figure 2d shows how the node execution is signified by a dark gray color, giving immediate processing feedback to the user. The events trigger algorithm execution only in nodes (or subsequent nodes) that require an update. This allows the algorithm development to only require a minimal recalculation for the algorithm element being modified and dependent downstream elements. This also provides a fast mechanism for analyzing the algorithm through diagnostic nodes that can be quickly assembled to manipulate and visualize data. Networks can be saved to a file for future use allowing full algorithms, algorithm components, or commonly used macros to be retrievable as needed. The networks can also be invoked in a nongraphical session effectively as a script from a terminal shell session. This allows users to run multiple reconstructions and change parameters where batch processing is needed. This also presents the potential for integration into other reconstruction software. Code Interfaces Each node on the canvas represents a Python interface to underlying algorithm code, which can be written in any language that can be bound to Python. The two languages currently used in GPI for numerical algorithm development are Python and Cþþ. Python is also used to set UI elements (i.e., widgets and ports) of the node interface. GPI Node Interface Node Interface (Python) GPI nodes are defined as a subclass of a generic GPI node object interface (NodeAPI) as shown in the code

1451

sample in Figure 3. This interface is where node widgets, input ports, output ports, and algorithm code is defined. This is also where help text and the node usage information may be defined as shown in the first comment under the ExternalNode class declaration. Help text defined here is extracted at runtime and placed in an About widget allowing the user easy access to this information from the UI without having to examine the code. The two class methods that must be implemented to create a node are the initUI() and compute() methods, which are shown in the remainder of the example. Widget and port declarations are placed in the initUI() function shown as the first method definition. To simplify the UI generation process, widgets and port definitions were designed to be parameterized with one line of code each. The UI layout for widgets is a vertical stack-ordered top to bottom graphically by the order in which they are defined in the node definition code. Ports are positioned from the left to right on the node graphic in the order in which they are declared as well. The widget macros and forced widget layout greatly simplify the UI code reducing the code burden on the developer. The node definition shows that widgets and ports are named and given specific types. The widget types (such as sliders and buttons) are predefined, however, there is a mechanism for defining new widgets. In the widget declaration, the widget type is chosen, then the widget is named, followed by any attributes specific to that widget, which are set as additional keyword arguments. Ports are declared in a similar manner requiring a name and the data-type that they will enforce on the canvas. Data enforcement is a mechanism for restricting the nodes input connections to ensure the correct operating conditions for the algorithm. Input and output ports that do not have the same data type specifications will not be allowed to connect, giving the user instant feedback on the compatibility of two algorithm elements. The second method definition (compute()) is where the algorithm is implemented. As the example in Figure 3 shows, the algorithm code is usually placed after widget and port “getter” functions such as getVal() and getData(). These functions return the user input data from widgets and data from upstream nodes. After the algorithm is complete, an output port is set using the setData() function and control is returned to the canvas. The compute() function terminates with a return statement which can be used to display warnings and errors depending on the numeric return code. Figure 2e and f shows a Shapes node in idle (light gray) and a Statistics node in both warning state (yellow) and error state (red). These two states tell the user if the node code has crashed or if it is improperly parameterized depending on how the developer implements the return codes. One of the more useful prototyping features is the ability to choose how the compute() routine will be executed, either as a process, a thread, or as part of the main execution loop. Processes can be easily terminated from the canvas by deleting a node that is in the middle of execution. Threads share the same memory space

1452

Zwart and Pipe

FIG. 3. Example code for a simple GPI node (top) and Python extension module code implemented with the Python Function Interface (PyFI; bottom).

with GPI, and therefore, provide faster data transfer between nodes and a lower memory footprint. Nodes that are executed as part of the main execution loop can implement other GUI packages, such as Matplotlib (13), that implement their own GUI framework. There is one other definable function called validate() that allows developers to write additional checks and modifications to the UI. This is usually used for menus that require updates based on incoming data, initial calculations, and other recurring changes that must be made to the UI. This function runs before compute() and is always run as part of the main execution loop. Since Python scripts can be imported as a library, the pure algorithm does not have to reside in the node definition file. The compute() method can simply call an externally defined function that does not contain any GPI-specific calls. While this is more of a coding practice than a GPI feature, it demonstrates how GPI can implement functionality from external packages and be used to develop code for other platforms cleanly.

Python Function Interface Module Interface Algorithms that are developed in Cþþ can also be integrated into GPI via the Python C-API. While the Python C-API provides unrestricted access to the inner workings of most aspects of Python, it can be daunting for implementing routine Cþþ algorithms on a regular basis. For this reason, a python function interface (PyFI) library was also developed to simplify this process for practical algorithm development. PyFI is intended to be a light-weight interface that provides customizable translations between Python and Cþþ data objects, similar in concept to the Boost–Python (14) and Simplified Wrapper and Interface Generator (SWIG) (15) projects. Most of the GPI nodes developed so far use the Numpy package (16), which is a popular array structure for scientific computing in Python. The main purpose of PyFI is to simplify the translation of Numpy arrays to Cþþ array objects (and vice versa) as well as minimize the code necessary to generate a Cþþ Python module extension in ways that are geared toward

Graphical Programming Interface

generating scientific code. Algorithm code written in Cþþ can be shared between GPI and other platforms using the array objects native to those platforms after PyFI has been customized to do the translation. The two requirements of generating sharable code from GPI is to package the code into modules and to use a common data translation interface. Algorithms written in pure-Python modules (without any GPI interface calls) can be shared directly with other Python users. For Cþþ code, a common data translation interface (i.e., any of the aforementioned interfaces) is required to translate the types of data passed between the modules and languages (i.e., integers, floats, strings, arrays, etc.). As an example of portable Cþþ code, some of the GPI nodes presented in this work encapsulate code that is capable of running on the Philips proprietary reconstruction platform. To make the algorithms portable between the two systems, array memory must be translated from the Numpy array, in Python, to the Philips reconstruction array library in Cþþ. This translation is done in a supplementary GPI library that is connected to PyFI. The code sample in Figure 3 shows how the PyFI macros simplify the function definition of a Cþþ function (myFunc()) that will be made accessible to Python. In this example, the PYFI_FUNC, PYFI_START, and PYFI_END macros inject code that handles the special Python function declaration, argument parsing, and memory cleanup. Positional function arguments are declared with one line of code (for all supported data types including arrays) using the PYFI_POSARG macro. This macro converts array arguments without copying data by encapsulating the Python memory segment in a PyFI Array object. Similar macros exist for parsing keyword arguments and for initiating Python side memory allocation. The output also only requires a single line of code to perform the reverse operation of translating the output array to a Numpy array using the PYFI_SETOUTPUT macro (which can be performed as a copy or direct use of preallocated Python memory). Multiple input and output arguments correspond to the order in which the macros are called. Portable Cþþ functions that operate on the input data and generate output data would then be called between the PYFI_POSARG and PYFI_SETOUTPUT macros allowing the function definition to be free of any GPI-specific code. The finished algorithm can then be easily deployed to another system (e.g., to a reconstruction processor) as it contains no GPI-specific code. Use Cases The following sections cover the experimental setup for highlighting aspects of different network designs and for tracking the code utilization in an existing GPI development environment. Execution Approaches The node execution options mentioned in the Node Interface section allow the developer to take advantage of the properties of each runtime. Running nodes as processes allow the developer to recompile and import extension modules without restarting the Python interpreter session as opposed to both thread and main loop

1453

execution. This allows changes to extension modules to be tested rapidly in-place over multiple compilations. Node-processes also isolate the main process (i.e., the GPI canvas) from failures in development code making the canvas robust to crashes caused by node code. In addition to the node-level processing choices, networks themselves can be developed with different tradeoffs. The memory footprint of a network can be minimized by allowing each node to process one data element (of a set) at a time. Conversely, the whole dataset can be passed to each node allowing the node to process all the data at the cost of using more memory. To illustrate these differences in combination with the node execution options, a 2D spiral reconstruction is performed using both network topologies. Each network is run with all nodes as separate processes or as one “main” process. Since the execution mode is typically chosen on a per node basis, forcing all of the nodes to one or another will provide bounds for the computational duration. Figure 4 shows two network configurations. A 2D spiral (17) phantom dataset acquired with 79 spiral interleaves, 825 samples per interleave, 1 channel, and 20 slices (for a total size of 10 MB) is reconstructed on a grid matrix of 300  300 elements. The network on the left (Net. #1) passes all the data (in this case, for 20 2D slices) directly to the Grid node which, in turn, passes all gridded data downstream for subsequent nodes to perform bulk processing. The network to the right (Net. #2) contains three extra nodes (Reduce, AutoNum, and Glue) which work to loop over the data stack and send individual slices through the subsequent nodes. At the end, the Glue node concatenates the reconstructed data segments or merges the final data to a preallocated segment. As an additional point of reference, the PROPELLER network was run using topologies similar to the Net. #1 and Net. #2 examples (not shown) for both process and main execution modes. The volunteer data was acquired with 23 PROPELLER blades, blade dimensions of 30  872 samples, 24 slices, and 12 channels (for a total size of 1.4 GB) and was reconstructed on a 436  436 grid matrix. Each network was executed on a 2.6 GHz Intel i7 processor. Code Utilization Code reuse is made simple through the graphical interface via widgets and ports. Every time a node is used it represents another instance of code reusage. To illustrate how code is used, the networks for 10 MR developers of the author’s lab were analyzed for usage statistics such as number of node instances, lines of code, and network sizes. The line counts for node code are split into two categories to determine the amount of overhead code that a developer must generate to implement an algorithm. The code required to define a node’s UI (i.e., the initUI() and validate() functions) is considered overhead while the code contained in the compute() routine, C/ Cþþ and Python extension modules are considered necessary algorithm code. The line counts for both Python and C/Cþþ code omit whitespace and comment lines.

1454

Zwart and Pipe

FIG. 4. Two 2D spiral reconstruction networks on the same canvas with a complex display. The network on left passes all 2D data sets for each node to reconstruct. The network on the right reconstructs data one slice at a time and concatenates the results. In the display window, the reconstructed image phase data is colorized and the brightness is modulated by the data magnitude.

The overhead percentage is calculated as the sum of all overhead lines for all nodes divided by the total number of lines for all nodes. A “reuse factor” for nodes and code is calculated to demonstrate how often implemented work is reused. The reuse factor for nodes is calculated as the number of node instances divided by the number of unique nodes. Similarly the code reuse factor is calculated as the number of lines times the number of instances divided by the number of unique node lines. The node instances are also binned in terms of total usage and categorized based on the functional nature such as visualization, file input and output, generic non-Cartesian, math and array manipulation functions, and application specific algorithms. RESULTS This section covers several example applications that illustrate how GPI may be used. The first section uses the PROPELLER (18–20) reconstruction network to introduce a non-Cartesian pipeline and some of the display elements available in GPI. The second shows a couple of spiral reconstructions that range in complexity and show how data analysis is made easier with visual programming. The third section gives an example of a spin simulator (a nonreconstruction problem) that is also wellsuited for exploration in a graphical environment. The final section covers use cases for execution and code utilization. PROPELLER Supporting Information Figure S1 shows a reprint of the PROPELLER reconstruction diagram from (18) next to a screenshot of the GPI PROPELLER network. The visual programming interface offers a direct relationship between the theory and implementation of the algorithm. The implementation of the algorithm is split up into similar modular elements, some with generic functionality and others specific to this application. Offline reconstruction applications start off with the same initial step of reading MR data in a file from the

scanner. The PROPELLER reconstruction network in Figure 5 shows the data reader as the highest level node called ReadPhilips. This node reads the vendorspecific format into a multidimensional array (a Python array object (16)) and passes it downstream for subsequent processing. In this example, the PROPELLOR_CoilCombine node is the next matching operation in the reprinted flow diagram. The coordinate generation, rotation estimation, and correction are all done in the PROPELLOR_Crds node. The rotation corrected coordinates and estimates are then made available as outputs. The data get passed down to the PROPELLOR_ShiftCorrect node which performs the translation estimation and correction on the data. The motion compensated coordinates get passed to the non-Cartesian density estimation node SDC. The coordinates, data, and density compensation are all sent to the gridding node to begin the generic non-Cartesian reconstruction segment. The naming scheme chosen for this network shows that nodes with names that are prefixed with PROPELLOR are specific to this application, however, the remaining nodes are generic array manipulators or generic nonCartesian reconstruction elements such as Grid for 2D gridding (21,22), FFTW implements the fast Fourier transform (23), Rolloff for grid deapodization and SDC for estimating sample density (24–27). A significant advantage of the graphical interface is that the data and algorithm can be probed at various points of execution because the out-ports of each node hold the result of the node algorithm. Figure 5 shows the PROPELLER network with additional display and array manipulation tools grabbing data from various points of interest and displaying them in a 2D image viewer. Starting from the top of the network, the coil images of a single PROPELLER blade are visualized as colorized phase maps with superimposed magnitude. Second, the kspace data are taken from after the Subsample node, cropped, and then transformed into image space to see each coil image with coil sensitivity modulation. Then the data after the coil combine operation are cropped

Graphical Programming Interface

1455

FIG. 5. The PROPELLER network with probes at points of interest.

and transformed into image space to reveal a single blade image without coil sensitivity modulation (as indicated by the lack of phase variation across the image). The kspace data for each blade can also be visualized by stacking all the blades in 2D before sending to the ImageDisplay node. The density compensation data is also displayed in a 2D viewing window showing that each blade is not weighted the same due the applied motion compensation. The motion compensation is also evident in the gridded k-space image where rotational motion has caused some of the blades to overlap and leave unsampled wedge-shaped locations in high frequency kspace. The final reconstructed image is shown in Figure 6a and b where the image viewer window is combined with two extra push button widgets below. This is an example of a composite layout window which can be constructed to hold only the elements of the network that are of interest when examining a network. In this case, the two buttons have been taken (via drag-and-drop) from the PROPELLER motion correction nodes and placed under the viewer that is displaying the final reconstruction result. In this example, the composite layout window is used to focus the user on the algorithm result in terms of the two input push-buttons that control the rotation and translation compensation. Figure 6a shows the buttons in the “off” state and the resulting image is motion

blurred. Figure 6b shows the buttons in the “on” state and the resulting image is motion compensated. Figure 6c shows an additional layout containing two plot widgets with the estimated translations (for x and y) and rotations. All of the data for these figures are easily pulled from the canvas since each of the nodes holds their copy of the data once the node has executed. While the reader node in this example is specific to the Philips proprietary format, there are currently GPI nodes that can read (and write) a variety of different formats such as .mat, .hdf5, .csv, .raw, .npy, .pickle, .jpg, and .png. The number of file formats available to developers in the MR community underscores the need for modularity in a common framework. This allows developers to contribute a reader (or writer) of a given format and gain all of the tools available in the common framework. The ability to import from a wide range of formats also allows developers to make comparisons between different systems.

Spiral The spiral k-space trajectory is another example of a non-Cartesian trajectory that can easily be reconstructed using the same gridding method shown in the PROPELLER section. This section will show a few examples of

1456

Zwart and Pipe

FIG. 6. Reconstructed PROPELLER images with motion correction a: off and b: on and plots of measured x-y shifts and rotational motion for each PROPELLER blade (c).

2D and 3D spiral implementations presented in Refs. [17,28,29). The GPI network for spiral reconstruction is shown in Figure 4. Similar to the PROPELLER network, the spiral reconstruction starts with a raw data file reader node (ReadPhilips). In this case, the raw data is passed to the gridding node and protocol parameters are passed to a spiral coordinate generator (SpiralCoords). The subsequent nodes are generic for non-Cartesian reconstruction. First, sample densities are estimated, then the gridding, FFT, and deapodization are executed to form the final image. Since most of the nodes are generic, the development time for creating a new application is reduced, any improvements in the common nodes will benefit all applications, and developers only need to maintain the node’s port interface to ensure that the node will work with other applications. Figure 4 also shows a snapshot of the ImageDisplay node’s widget menu, which contains options for displaying complex input data as a real, imaginary, magnitude, phase, and colorized phase composite image. The composite image is shown with a color wheel surrounding the image with cyan representing a zero-phase, green   and magenta representing þ90 and 90 , and red repre senting 180 . The last widget shown is a group of sliders that control the image scale using the common levelwindow and floor-ceiling conventions.

CG Deblurring GPI naturally makes comparing different reconstruction algorithms a simple task. The spiral fat–water separation and simultaneous deblurring network is shown in Figure 7 is an example of how algorithm development can fork, from a larger reconstruction, and be simultaneously constructed for comparison. In this case, the two algorithms being compared follow the same node topology so the

sections labeled multipeak and single-peak are mostly duplicates. The user only needs to use the copy–paste functionality, from the canvas, to instantiate the duplicate nodes. The BlurKernel node contains an algorithm that generates either a single-peak spectral fat model or a multipeak model. The result of both paths funnel into a Combine node, which takes the output fat and water images and places them side-by-side for displaying. The two reconstruction paths make use of a generic iterative conjugate gradient node that simply takes the initial conditions and iterates over a cyclic node connection, outputting the solution, and residual at each iteration. This cyclic connection is made through the Deblur_FW node, which is simultaneously connected downstream from the ConjugateGradient node (black edge) and upstream (red edge). Solutions from each iteration can be stored using a Glue node (not shown), which expands its internal array buffer as new data are passed to its input port. Figure 7 shows solution images taken from the first iteration (n ¼ 1) and eighth iteration (n ¼ 8) for both fat and water images labeled “F” and “W,” respectively. Signal measurements from the images can be made by extracting a region-of-interest using the Reduce and Statistics nodes (Results shown in (30)).

Trajectory and Gradient Waveforms As mentioned earlier, the GPI framework is not specific to running MR reconstructions. k-Space trajectories that are developed using the Cþþ extension interface can be implemented on the scanner platform (provided it supports C/Cþþ) as well as on the reconstruction side. GPI is used to prototype the k-space trajectory and gradient waveforms using visualization tools such as a 2D plotter and in 3D using OpenGL rendering (31). The screenshot shown in Supporting Information Figure S2 shows the SpiralCoords node generating a FLORET 3D spiral trajectory (29). The SpiralCoords node

Graphical Programming Interface

1457

FIG. 7. A screenshot of a spiral reconstruction network with simultaneous fat-water separation and spiral deblurring processes. Two methods of spiral fat–water/deblurring are compared (a multipeak vs. single-peak fat spectral model).

also generates the corresponding gradient waveforms to the trajectory. The 2D x-y spiral coordinates and individual gradient channels are readily available for visualization in the plotter node and the 3D spiral coordinates are converted to Graphics Library (GL) objects for rendering in the GLViewer node. These tools are used to visualize the sequencing and sampling patterns while developing new k-space trajectories.

Spin Simulator Spin simulation is another application that is well-suited for GPI. The spin simulation network shown in Figure 8 numerically solves the Bloch equations for a configurable spin array. The spin array (generated within the Spyn node) contains information about the spatial position, velocity, density, resonant frequency, relaxation rates, initial position, and solution time step-size. Each timestep is solved in the Bloch node for a given duration, gradient, and RF waveforms. As shown, the Bloch nodes can be concatenated to produce different, time varying, gradient, and RF sequences. Three helper nodes (MRPSD1D, MRSig1D, and SpinVis3D) are used to extract and reformat various pieces of the spin array for visualization. The MRPSD1D node extracts the gradient and RF waveform arrays for plotting. The MRSig1D node extracts transverse and longitudinal magnetization for plotting against a selected ordinate axis (e.g., time, T1, T2, frequency, position, or velocity). The example plot shows transverse magnetization in x and y (Mx and My) versus

time. The SpinViz3D node creates a 3D scene of spatially distributed spins (represented as spheres) with relative moment represented by a protruding cylinder. The scene can be updated at each time index and the spins can be colorized by relative phase, magnetization transfer, and position. Tracers can also be added to the cylinder tips to give a sense of the spin evolution over time as shown in the 3D viewer window. Use Cases The following sections present the results of the different aspects of GPI usage both in terms of network design and code utilization. Execution Approaches The different execution modes combined with the two network topologies shown in Figure 4 produced the memory footprints and computational durations shown in Table 1. As the results for Net. #2 show, passing the individual slices results in a lower memory footprint since each node only holds onto the last processed slice. However, if all the nodes are run as processes with this topology, the processing overhead increases due to the number of times each node is executed (i.e., by the number of slices in this dataset). The Net. #1 approach increases the memory footprint as all slices at each stage are kept, but the computational duration is lower due to the fewer number of spawned processes. Running nodes in the main execution mode shows a significant

1458

Zwart and Pipe

FIG. 8. A screenshot of a spin simulator network with RF and gradient waveform plotting, transverse magnetization versus time plots and a 3D visualization of the spin evolution in time.

reduction in computational time for Net. #2 and moderate reduction for Net. #1. This process mode overhead for the PROPELLER networks is less disparate due to the relatively more complex underlying algorithms (requiring more computational time relative to the process overhead). It is likely that the process mode will be used for its development advantages during algorithm prototyping, while the computational efficiency of the other modes are more likely to be used in a batch processing capacity. Code Utilization Table 2 shows the results of the analysis performed on 178 networks from 10 MR developers using GPI. The subtable under the “frequency of node use” label is essentially a histogram of node usage. This shows that more generic nodes, such as those used in array math or manipulation, are used more often than those with a specific application. The “File I/O” and “Visualization” categories represent this trend. In terms of file formats, the Numpy and Philips proprietary formats are currently the most common in the context of this lab. However, each of the other implemented formats are also used. For visualization, the plotter node and 2D viewer nodes are the

most common as they naturally represent the basics of data visualization while the use of the 3D GL rendering nodes are used primarily for a relatively smaller set of spin simulations in this case. Since visualization nodes are often used during the prototyping phase as a debugging tool, they are not routinely saved in networks, and therefore, the relative number of instances reported is lower than the frequency with which they are actually used. However, they are often used as the final output (in addition to file I/O nodes) and are present in 87% of the networks. Since the purpose of networks can range from simple file format translation to full MR reconstruction Table 1 Computational Durations and Memory Footprint for Network Topology and Execution Mode Options Processes Net. #1 Net. #2 PROPELLER #1 PROPELLER #2

3.5 s 18.87 s 3.5 min 4.9 min

Main 1.2 3.7 2.6 3.0

s s min min

Total memory 89.1 MB 31.58 MB 3.66 GB 1.48 GB

Nodes were executed either as a separate “process” or as part of one “main” process.

Graphical Programming Interface

1459

Table 2 Node Code Statistics Source data # of users # of networks

10 178

Node statistics # of unique nodes # of node instances # of C/Cþþ nodes # of pure Python nodes Average # of nodes/network Standard deviation # of nodes/network Max # of nodes/network Min # of nodes/network Node reuse factor Frequency of node use: # of nodes 30 # of instances

1

163 5,441 50 113 31 31 148 1 33

86

30

17

2  10 11  100 101  1000

Category instances : Visualization

0

8

2

1

File I=O

4

10

3

3

Non-Cartesian

10

26

4

3

Math=manipulation

7

13

10

10

App specific

9

29

11

0

Visualization nodes # of unique visualization nodes Percentage of networks Containing visualization nodes

11 87%

Code statistics Total lines of node code Total lines weighted by instance Average # of lines/node Standard deviation # of lines/node Percentage of overhead: All nodes C/Cþþ nodes Pure Python nodes Code reuse factor

pipelines the number of nodes per network varies significantly. Table 2 shows that the minimum network size is 1 node which represents the use of saving preparameterized nodes. Nodes that can be used to define temporary code via a text-box widget are often saved as a macro for provisional or initial implementations. The code statistics shown toward the bottom of Table 2 indicate that, similar to the networks, the nodes themselves can vary significantly in complexity with an average line count of 339 and a standard deviation of 603 lines. The code overhead is shown to be 4.6% for the average C/Cþþ node and 27.9% for the average Python node. This is assumed to be due to the relatively compact nature of Python code versus C/Cþþ. The relative code reuse factor of 32 closely matches the node reuse factor of 33 indicating a high level of utilization for a developer’s efforts on average.

55,227 1,764,187 339 603 11.8% 4.6% 27.9% 32

DISCUSSION The GPI built-in node libraries provide tools for nonCartesian MR reconstruction, spin simulation, and data visualization. In addition to ongoing development of the built-in node libraries, it is the purpose of this project to minimize the effort required to develop new techniques by continuing to develop and simplify the framework. Since the framework is not MR-specific, its application is intended to be library driven. GPI provides a simple interactive way for developers to probe their algorithms at any point along the network. The trade-off for this aspect of the tool is a substantial increase in required system memory. While this is not a problem for most 2D and 3D reconstructions and the examples shown, it may be a consideration for techniques that are already pushing the limits of currently available hardware.

1460

The GPI framework does not provide concurrent operation for nodes. However, parallel processing (including GPU processing) can be implemented within nodes. The specific package or technique is open for the developer to choose. Some of the future works planned for this project include continued expansion of the built-in node library and framework changes to address the mentioned drawbacks. The minimum system memory may be reduced by allowing the user to toggle ports to either hold on or free their memory after dependent nodes are finished executing. Adding node concurrency will also allow users to graphically initiate multiprocessing without having to implement a specific library programmatically. CONCLUSION The GPI framework is shown to be a versatile prototyping environment for developing numeric algorithms and scientific applications. Several MR applications, not limited to reconstruction, are demonstrated and highlight the properties of the modular framework. The graphical interface allows users to readily reuse existing modules, minimizing development time, and allows users to rapidly manipulate and visualize data. The code interface is shown to be simple, minimizing the barrier to implementing new algorithms and also provides the necessary level of complexity required in developing advanced MR techniques. REFERENCES 1. Hansen MS, Sørensen TS. Gadgetron: an open source framework for medical image reconstruction. Magn Reson Med 2013;69:1768–1776. 2. Grissom WA. Web-based, platform-independent dissemination of image reconstruction & RF pulse design algorithms using the google chrome native client. In: ISMRM Workshop on Data Sampling and Image Reconstruction. Sedona, Arizona, 2013; Session 8. 3. Uecker M. Software toolbox & programming library for compressed sensing & parallel imaging. In: ISMRM Workshop on Data Sampling and Image Reconstruction. Sedona, Arizona, 2013; Session 8. 4. Vahedipour K. Codeare—common data exchange & reconstruction framework. In: ISMRM Workshop on Data Sampling and Image Reconstruction. Sedona, Arizona, 2013; Session 8. 5. Block KT. Prototypic setup for evaluation of a compressed-sensing technique in clinical patient studies. In: ISMRM Workshop on Data Sampling and Image Reconstruction. Sedona, Arizona, 2013; Session 8. 6. Zenge MO. IRecon—introducing a standardized interface into the Siemens image reconstruction environment. In: ISMRM Workshop on Data Sampling and Image Reconstruction. Sedona, Arizona, 2013; Session 8. 7. Zwart NR, Pipe JG. Graphical programming interface. In: ISMRM Workshop on Data Sampling and Image Reconstruction. Sedona, Arizona, 2013; Session 8. 8. Zwart NR, Pipe JG. Graphical programming interface. Available at: http://gpilab.com. Accessed June 24, 2014. 9. Upson C, Faulhaber TA, Jr, Kamins D, Laidlaw D, Schlegel D, Vroom J, Gurwitz R, Van Dam A. The application visualization system: a computational environment for scientific visualization. IEEE Comput Graph Appl 1989;9(4):30–42.

Zwart and Pipe 10. van Rossum G, Drake FL. The Python Language Reference Manual. Available at: http://python.org. Accessed June 25, 2014. 11. Sanner MF, Stoffler D, Olson AJ. ViPEr, a visual programming environment for Python. In: Proceedings of the 10th International Python conference, Alexandria, Virginia, USA, 2002. p. 103–115. 12. Summerfield M. Rapid GUI programming with Python and Qt: the definitive guide to PyQt programming. Ann Arbor, MI: Pearson Education, 2007. 13. Hunter JD. Matplotlib: A 2D graphics environment. Comput Sci Eng 2007;9:90–95. 14. Abrahams D, Grosse-Kunstleve RW. Building hybrid systems with Boost.Python. C/Cþþ Users J 2003;21(LBNL–53142). 15. Beazley DM. SWIG: an easy to use tool for integrating scripting languages with C and Cþþ. In: Proceedings of the 4th USENIX Tcl/Tk workshop, Monterey, California, USA, 1996. p. 129–139. 16. Oliphant TE. Python for scientific computing. Comput Sci Eng 2007; 9:10–20. 17. Pipe JG, Zwart NR. Spiral trajectory design: a flexible numerical algorithm and base analytical equations. Magn Reson Med 2014;71:278–285. 18. Pipe JG, Gibbs WN, Li Z, Karis JP, Schar M, Zwart NR. Revised motion estimation algorithm for PROPELLER MRI. Magn Reson Med 2013;72:430–437. 19. Pipe JG. Motion correction with PROPELLER MRI: application to head motion and free-breathing cardiac imaging. Magn Reson Med 1999;42:963–969. 20. Forbes KP, Pipe JG, Bird CR, Heiserman JE. PROPELLER MRI: clinical testing of a novel technique for quantification and compensation of head motion. J Magn Reson Imaging 2001;14:215–222. 21. Jackson JI, Meyer CH, Nishimura DG, Macovski A. Selection of a convolution function for Fourier inversion using gridding [computerised tomography application]. IEEE Trans Med Imaging, 1991;10:473–478. 22. Beatty PJ, Nishimura DG, Pauly JM. Rapid gridding reconstruction with a minimal oversampling ratio. IEEE Trans Med Imaging 2005;24: 799–808. 23. Frigo M, Johnson SG. The design and implementation of FFTW3. Proc IEEE 2005;93:216–231. 24. Zwart NR, Johnson KO, Pipe JG. Efficient sample density estimation by combining gridding and an optimized kernel. Magn Reson Med 2012;67:701–710. 25. Johnson KO, Pipe JG. Convolution kernel design and efficient algorithm for sampling density correction. Magn Reson Med 2009;61: 439–447. 26. Pipe JG, Menon P. Sampling density compensation in MRI: rationale and an iterative numerical solution. Magn Reson Med 1999;41:179– 86. 27. Pipe JG. Reconstructing MR images from undersampled data: dataweighting considerations. Magn Reson Med 2000;43:867–875. 28. Turley DC, Pipe JG. Distributed spirals: a new class of threedimensional k-space trajectories. Magn Reson Med 2013;70:413–419. 29. Pipe JG, Zwart NR, Aboussouan EA, Robison RK, Devaraj A, Johnson KO. A new design and rationale for 3D orthogonally oversampled kspace trajectories. Magn Reson Med 2011;66:1303–1311. 30. Zwart NR, Wang DH, Pipe JG. Spiral CG deblurring and fat-water separation using a multi-peak fat model. In: Proceedings of the 22nd Annual Meeting of ISMRM. Milan, Italy, 2014. p. 1660. 31. Fletcher M, Liebscher R. PyOpenGL, the Python OpenGL binding. Available at: http://pyopengl.sourceforge.net. Accessed on June 25, 2014.

SUPPORTING INFORMATION Additional Supporting Information may be found in the online version of this article. Supporting Figure S1: The PROPELLER reconstruction graph (a) previously published in (18) along side the GPI implementation (b) Supporting Figure S2: A screenshot of the 3D spiral FLORET coordinate and gradient waveform generator node with 2D and 3D plots of the trajectory and gradient waveforms