Linking Chimera with our molecular simulation tool-kit

Dear Chimera developers, I am a postdoc research associate working with Prof. Lev Kantorovich of King’s College London. I am currently a full time developer for the ab initio atomic simulation code CP2K, and as a part of our project funding, we are looking to develop a tool kit that would improve user friendliness that would allow the users to construct various bulk or molecular structures in an intuitive and graphically way. We currently have a quite extensive tool-kit (tetr) developed by Prof. Kantorovich for generating, modifying structures—especially tailored for periodic systems, as well as post processing utilities on volume data (such as generating STM images etc). It is written in FORTRAN, and currently is command based only, with the user input being either interactive or by a batch file. The goal, is to be able to link to a visualisation package like Chimera, so that: a) one is able to automatically update the visualisation when the user uses the tool interactively b) allow the user to select atoms and fragments and use these selections as an input to the tool-kit. Ideally, the took-kit can run as an independent code, with the extension providing an interface between Chimera and the tool-kit. The user would then control the tool-kit from within the Chimera command interface. I have used Chimera a lot during my PhD, however, I have not yet written any extension for it. I do, however, have quite a number of years experiences with code development, especially in Fortran and in Python. I am wondering: 1) If you would mind if we develop an extension to link our took-kit with Chimera 2) If the functionalities I mentioned above would be possible to achieve Any suggestions from you would be great. Thank you very much for your time, and I look forward to hearing from you soon. Best regards, Lianheng ==================================== Lianheng Tong Room S4.02, Strand Building Department of Physics Kings College London Strand, London WC2R 2LS, U.K. Tel: +44 20 7848 2064 Fax: +44 20 7848 2420 lianheng.tong@kcl.ac.uk ====================================

Hi Lianheng, One of our design goals for Chimera was to enable the development of extension for third parties, so obviously we’d be happy if you made an extension for running CP2K via Chimera. I believe all the capabilities you outline are available in Chimera. The first thing I’d do if I were you is to look at the Chimera Programmer’s Guide, particularly the Examples. They don’t have enough detail to get you completely going, but they do cover the basic concepts of how things work under the hood in Chimera. Links: http://www.cgl.ucsf.edu/chimera/docs/ProgrammersGuide/index.html <http://www.cgl.ucsf.edu/chimera/docs/ProgrammersGuide/index.html> http://www.cgl.ucsf.edu/chimera/docs/ProgrammersGuide/Examples/index.html <http://www.cgl.ucsf.edu/chimera/docs/ProgrammersGuide/Examples/index.html> The next thing would be to look at a couple of extensions that do things along the lines of what you want to do. The ones that come to my mind are the tools in the “Amber” tools category (Add Ions, Solvate, Write Prmtop) that present some kind of interface to the user and, based on the interface settings, deliver commands to various Amber executables on the back end, forward the text output of the commands to the reply log, and open the output files (for Solvate and Add Ions) and present their structures in the graphics window. The code for these extensions are in <your Chimera app>/share/tool-name, so tool-name is either Solvate, WritePrmtop, or Addions. As detailed the Programmer’s Guide, each of these directories have a ChimeraExtension.py file, which contains code to tell Chimera where to put the tool in the Tools menu, and optionally what command to add for the tool, what file types it’s a reader for, and what button to add to the Model Panel for it. These tools put their non-GUI code into __init__.py and their GUI code in gui.py. This segregation makes it easier to offer a command version of the tool. To see how the tool drives the backend executable look at __init__.py for Solvate for example, in particular the initiateSolvate function. In general terms, that routine writes the input files that will be read by Amber’s sleap executable, forms the sleap command it wants to send, use’s Python’s subprocess.Popen to run the command, send the input command, and collect the output. As the command executes, it’s output is forwarded line by line to both the status line and the reply log. The routine then checks for the existence of the expected output file and raises an error if it’s missing. Otherwise it goes ahead and processes the file. The processing is pretty complicated as it tries to find the correspondences between the input, which has solvent added, and the original structure and I recommend you ignore that part of the code! It’s not 100% clear from your mail if your desired expected interaction between Chimera and CP2K fits into this scheme of one-shot (or series of one-shot) executions of CP2K, or if you were thinking of some kind of “stateful” execution where Chimera stays in continuous communication with one instance of CP2K. The latter is also possible but in the interests of not turning this mail into a George R.R. Martin novel, I'll cover that if you ask about it. :-) One of the points in your mail was getting info about what’s selected. It’s reasonably simple: from chimera.selection import currentAtoms selAtoms = currentAtoms() There are various attributes of Atom you may be interested in, such as name, element, residue, coord() and/or xformCoord(), and attributes of Residue such as type, id.position, id.chainId, id.insertionCode. You may want to use help() on these classes (chimera.Atom, chimera.Residue) in the Python shell (Tools->General Controls->IDLE) to get more extensive info. Lastly, asking on this mailing list is probably the quickest way to get turnaround on problems you run into. We’re happy to help! —Eric Eric Pettersen UCSF Computer Graphics Lab
On Oct 21, 2015, at 5:36 AM, Lianheng Tong <lianheng.tong@kcl.ac.uk> wrote:
Dear Chimera developers,
I am a postdoc research associate working with Prof. Lev Kantorovich of King’s College London. I am currently a full time developer for the ab initio atomic simulation code CP2K, and as a part of our project funding, we are looking to develop a tool kit that would improve user friendliness that would allow the users to construct various bulk or molecular structures in an intuitive and graphically way.
We currently have a quite extensive tool-kit (tetr) developed by Prof. Kantorovich for generating, modifying structures—especially tailored for periodic systems, as well as post processing utilities on volume data (such as generating STM images etc). It is written in FORTRAN, and currently is command based only, with the user input being either interactive or by a batch file. The goal, is to be able to link to a visualisation package like Chimera, so that:
a) one is able to automatically update the visualisation when the user uses the tool interactively b) allow the user to select atoms and fragments and use these selections as an input to the tool-kit.
Ideally, the took-kit can run as an independent code, with the extension providing an interface between Chimera and the tool-kit. The user would then control the tool-kit from within the Chimera command interface.
I have used Chimera a lot during my PhD, however, I have not yet written any extension for it. I do, however, have quite a number of years experiences with code development, especially in Fortran and in Python. I am wondering:
1) If you would mind if we develop an extension to link our took-kit with Chimera 2) If the functionalities I mentioned above would be possible to achieve
Any suggestions from you would be great. Thank you very much for your time, and I look forward to hearing from you soon.
Best regards,
Lianheng
==================================== Lianheng Tong Room S4.02, Strand Building Department of Physics Kings College London Strand, London WC2R 2LS, U.K. Tel: +44 20 7848 2064 Fax: +44 20 7848 2420 lianheng.tong@kcl.ac.uk <mailto:lianheng.tong@kcl.ac.uk> ====================================
_______________________________________________ Chimera-dev mailing list Chimera-dev@cgl.ucsf.edu http://plato.cgl.ucsf.edu/mailman/listinfo/chimera-dev

Hi Eric, Thank you very much for your prompt reply and help. In terms of the interface, sorry for not have been very clear earlier. There are actually two code packages in question: 1. CP2K: the main molecular dynamics simulation code 2. tetr: a tool that is used to construct and modify initial atomic structures which are used as inputs for CP2K. The initial plan is to first link Chimera with tetr. So that any geometry construction and manipulation can be viewed in real-time within the Chimera window, and any atomic selections within Chimera is automatically passed into tetr. In this case the communication between tetr and Chimera ideally should be continuous, or at least, auto update every 1 second intervals. So any more information on how I may achieve this will be of great help. One way I am currently thinking of doing is to modify tetr into a library package—which shouldn’t be too difficult, and would benefit future use of the code anyway---and then build a python interface for it. This way, the interactive prompt can be built directly into the Chimera extension, and I won’t have to worry about doing continuous communications between Chimera and tetr. ------------- Now that you have mentioned CP2K, I think it is also a very nice idea to may be link Chimera with CP2K, like the way Amber executables are linked. This way, full density functional theory calculations may be performed directly within Chimera window, and this would allow quantum mechanical geometry relaxations, viewing ground state electron charge densities, generate STM images etc, without the user having to leave Chimera. In this case, I think the way one can achieve this is to: 1. Using the tetr interface to generate an input file for CP2K calculation 2. Use Chimera to run CP2K in a standard Linux shell, with however number of processors defined by the user 3. After CP2K has finished execution, read in the relevant output from CP2K back into Chimera, for example, the xmol movie file for geometry relaxation, molecular dynamics, or electronic density files etc. 4. Chimera can then visualise those data in the normal way In other words, it will be a one-shot or several-shot execution, and continuous interaction between Chimera and CP2K won’t be required. Nevertheless, I think it makes sense for me to do this after the tetr interface. ------------- tetr is a freeware and open source, and CP2K is an open source code with GPL license (http://www.cp2k.org). I hope there won’t be any licensing issues I need to be careful about? ------------- I am sure I will have many questions to ask you and other developers when I start working on the tetr interface. Thank you very much for your help again. Best regards, Lianheng ==================================== Lianheng Tong Room S4.02, Strand Building Department of Physics Kings College London Strand, London WC2R 2LS, U.K. Tel: +44 20 7848 2064 Fax: +44 20 7848 2420 lianheng.tong@kcl.ac.uk ====================================
On 21 Oct 2015, at 21:54, Eric Pettersen <pett@cgl.ucsf.edu> wrote:
Hi Lianheng, One of our design goals for Chimera was to enable the development of extension for third parties, so obviously we’d be happy if you made an extension for running CP2K via Chimera. I believe all the capabilities you outline are available in Chimera. The first thing I’d do if I were you is to look at the Chimera Programmer’s Guide, particularly the Examples. They don’t have enough detail to get you completely going, but they do cover the basic concepts of how things work under the hood in Chimera. Links:
http://www.cgl.ucsf.edu/chimera/docs/ProgrammersGuide/index.html http://www.cgl.ucsf.edu/chimera/docs/ProgrammersGuide/Examples/index.html
The next thing would be to look at a couple of extensions that do things along the lines of what you want to do. The ones that come to my mind are the tools in the “Amber” tools category (Add Ions, Solvate, Write Prmtop) that present some kind of interface to the user and, based on the interface settings, deliver commands to various Amber executables on the back end, forward the text output of the commands to the reply log, and open the output files (for Solvate and Add Ions) and present their structures in the graphics window. The code for these extensions are in <your Chimera app>/share/tool-name, so tool-name is either Solvate, WritePrmtop, or Addions. As detailed the Programmer’s Guide, each of these directories have a ChimeraExtension.py file, which contains code to tell Chimera where to put the tool in the Tools menu, and optionally what command to add for the tool, what file types it’s a reader for, and what button to add to the Model Panel for it. These tools put their non-GUI code into __init__.py and their GUI code in gui.py. This segregation makes it easier to offer a command version of the tool. To see how the tool drives the backend executable look at __init__.py for Solvate for example, in particular the initiateSolvate function. In general terms, that routine writes the input files that will be read by Amber’s sleap executable, forms the sleap command it wants to send, use’s Python’s subprocess.Popen to run the command, send the input command, and collect the output. As the command executes, it’s output is forwarded line by line to both the status line and the reply log. The routine then checks for the existence of the expected output file and raises an error if it’s missing. Otherwise it goes ahead and processes the file. The processing is pretty complicated as it tries to find the correspondences between the input, which has solvent added, and the original structure and I recommend you ignore that part of the code! It’s not 100% clear from your mail if your desired expected interaction between Chimera and CP2K fits into this scheme of one-shot (or series of one-shot) executions of CP2K, or if you were thinking of some kind of “stateful” execution where Chimera stays in continuous communication with one instance of CP2K. The latter is also possible but in the interests of not turning this mail into a George R.R. Martin novel, I'll cover that if you ask about it. :-) One of the points in your mail was getting info about what’s selected. It’s reasonably simple:
from chimera.selection import currentAtoms selAtoms = currentAtoms()
There are various attributes of Atom you may be interested in, such as name, element, residue, coord() and/or xformCoord(), and attributes of Residue such as type, id.position, id.chainId, id.insertionCode. You may want to use help() on these classes (chimera.Atom, chimera.Residue) in the Python shell (Tools->General Controls->IDLE) to get more extensive info. Lastly, asking on this mailing list is probably the quickest way to get turnaround on problems you run into. We’re happy to help!
—Eric
Eric Pettersen UCSF Computer Graphics Lab
On Oct 21, 2015, at 5:36 AM, Lianheng Tong <lianheng.tong@kcl.ac.uk> wrote:
Dear Chimera developers,
I am a postdoc research associate working with Prof. Lev Kantorovich of King’s College London. I am currently a full time developer for the ab initio atomic simulation code CP2K, and as a part of our project funding, we are looking to develop a tool kit that would improve user friendliness that would allow the users to construct various bulk or molecular structures in an intuitive and graphically way.
We currently have a quite extensive tool-kit (tetr) developed by Prof. Kantorovich for generating, modifying structures—especially tailored for periodic systems, as well as post processing utilities on volume data (such as generating STM images etc). It is written in FORTRAN, and currently is command based only, with the user input being either interactive or by a batch file. The goal, is to be able to link to a visualisation package like Chimera, so that:
a) one is able to automatically update the visualisation when the user uses the tool interactively b) allow the user to select atoms and fragments and use these selections as an input to the tool-kit.
Ideally, the took-kit can run as an independent code, with the extension providing an interface between Chimera and the tool-kit. The user would then control the tool-kit from within the Chimera command interface.
I have used Chimera a lot during my PhD, however, I have not yet written any extension for it. I do, however, have quite a number of years experiences with code development, especially in Fortran and in Python. I am wondering:
1) If you would mind if we develop an extension to link our took-kit with Chimera 2) If the functionalities I mentioned above would be possible to achieve
Any suggestions from you would be great. Thank you very much for your time, and I look forward to hearing from you soon.
Best regards,
Lianheng
==================================== Lianheng Tong Room S4.02, Strand Building Department of Physics Kings College London Strand, London WC2R 2LS, U.K. Tel: +44 20 7848 2064 Fax: +44 20 7848 2420 lianheng.tong@kcl.ac.uk ====================================
_______________________________________________ Chimera-dev mailing list Chimera-dev@cgl.ucsf.edu http://plato.cgl.ucsf.edu/mailman/listinfo/chimera-dev

On Oct 22, 2015, at 8:07 AM, Lianheng Tong <lianheng.tong@kcl.ac.uk> wrote:
Hi Eric,
Thank you very much for your prompt reply and help.
In terms of the interface, sorry for not have been very clear earlier. There are actually two code packages in question:
1. CP2K: the main molecular dynamics simulation code 2. tetr: a tool that is used to construct and modify initial atomic structures which are used as inputs for CP2K.
The initial plan is to first link Chimera with tetr. So that any geometry construction and manipulation can be viewed in real-time within the Chimera window, and any atomic selections within Chimera is automatically passed into tetr. In this case the communication between tetr and Chimera ideally should be continuous, or at least, auto update every 1 second intervals. So any more information on how I may achieve this will be of great help.
One way I am currently thinking of doing is to modify tetr into a library package—which shouldn’t be too difficult, and would benefit future use of the code anyway---and then build a python interface for it. This way, the interactive prompt can be built directly into the Chimera extension, and I won’t have to worry about doing continuous communications between Chimera and tetr.
There are a lot of upsides to the library approach, but one tricky part is compiling your library in such a way that Chimera will be able to dynamically load it on the various versions of each OS (that you support) that users will have. You may have to use the same compiler and compiler version that Chimera uses, possibly along with various other settings. We would certainly help out if you go that route. Another option is to use Chimera as the “back end” to tetr. You would have tetr start Chimera along with Chimera’s REST interface (by supplying the shell args “—start RESTServer”) and examine Chimera's startup output for the port to contact the REST server on. REST communication is limited to normal Chimera commands, not Python, but you will receive notifications of models opening and closing, and changes in the selection. The “list” command can be used to list currently open models, the contents of the current selection, and a variety of other things. My guess is that being limited to the normal Chimera command set will not quite cut it for you, but I could be wrong. The documentation for RESTServer is here: http://www.cgl.ucsf.edu/chimera/current/docs/ContributedSoftware/restserver/... <http://www.cgl.ucsf.edu/chimera/current/docs/ContributedSoftware/restserver/...> My best guess as what the best option in your scenario is, is that you basically use the one-shot case but don’t close the Popen to the backend tetr process. Something in tetr’s output would need to mark that the output for a given command is complete (or the extension would have to know what, if anything, it cares about in the output and only read that far). When another command needs to go to tetr, it’s just added to the Popen stdin, and then the output examined again for whatever info is desired, etc.
-------------
Now that you have mentioned CP2K, I think it is also a very nice idea to may be link Chimera with CP2K, like the way Amber executables are linked. This way, full density functional theory calculations may be performed directly within Chimera window, and this would allow quantum mechanical geometry relaxations, viewing ground state electron charge densities, generate STM images etc, without the user having to leave Chimera. In this case, I think the way one can achieve this is to:
1. Using the tetr interface to generate an input file for CP2K calculation 2. Use Chimera to run CP2K in a standard Linux shell, with however number of processors defined by the user 3. After CP2K has finished execution, read in the relevant output from CP2K back into Chimera, for example, the xmol movie file for geometry relaxation, molecular dynamics, or electronic density files etc. 4. Chimera can then visualise those data in the normal way
In other words, it will be a one-shot or several-shot execution, and continuous interaction between Chimera and CP2K won’t be required.
Nevertheless, I think it makes sense for me to do this after the tetr interface.
I don’t know how long CP2K computations typically take, but if they typically take more than a minute you may need to run them in a separate thread, or via Chimera’s task manager, the latter of which would allow the user to quit Chimera and still keep running CP2K (or even save a session and come back later).
-------------
tetr is a freeware and open source, and CP2K is an open source code with GPL license (http://www.cp2k.org). I hope there won’t be any licensing issues I need to be careful about?
Our interpretation of GPL and GPL-like licenses is that if the software is a library that Chimera has to link against, even dynamically, then GPL would also apply to Chimera itself, which would be problematic (though LGPL would be okay). If the software is a standalone executable that Chimera communicates with via interprocess communication, then the GPL license applies to just the standalone software (so source code for such executables would have to be openly available). All this applies only to software we ship with Chimera. Software/extensions provided by third parties on their own would not affect Chimera proper in any way.
-------------
I am sure I will have many questions to ask you and other developers when I start working on the tetr interface.
Yep! :-) —Eric
participants (2)
-
Eric Pettersen
-
Lianheng Tong