- Chimera-users - RBVI Mailing Lists

Re: [Chimera-users] using a marker file to zone a volume from python prompt
by Thomas Goddard 30 Jan '07

30 Jan '07

Hi Jeff, I added a user interface dialog for rotating and translating atom coordinates. Markers in Chimera are equivalent to atoms so it works on those. The rotation is specified in Euler angles. The new dialog, called Transform Molecule Coordinates, is available on the experimental features web page: http://www.cgl.ucsf.edu/chimera/experimental/experimental.html Tom Jeff wrote: > thanks Tom, > > much appreciated. how hard is it for me to write a dialog GUI to provide > 3 numerical inputs to the rotation script you forwarded me? (is it more > involved than a few lines? I am not familiar with Tkinter). > > don't want to bother you too much, if it is not a quick thing/response > feel free to punt, please. > > -jeff >

1 0

Update volume display without pressing show
by Thomas Goddard 30 Jan '07

30 Jan '07

Hi Jeff, The volume dialog data display option "Update display without pressing Show" only effect volume dialog features. It does not have any effect on the Volume Planes tool. The Volume Planes tool does not allow adjusting the plane number without updating the display. Tom Jeff wrote: > hi Tom, > > I have volume planes and precomputed subsamplings going, and the 'update > display without pressing Show' no longer works (keep toggling on and > off, but it doesn't have an effect) > > is there a way for me to 'refresh' the state of buttons like that? > > -Jeff

1 0

Re: [Chimera-users] Representation of volume data in memory
by Thomas Goddard 30 Jan '07

30 Jan '07

Hi Jeff, I suspect 2-byte integers are not faster to access or compute with then 4-byte floats on modern CPUs. But more of the map will fit into the CPUs cache which could improve bandwidth and could make computing a contour surface faster. But I don't think that will be significant. You would need to test it to know. Chimera keeps the native data type of the map to conserve memory when reading large maps. Tom Jeff wrote: > does this mean that a 2-byte integer format of a file will display > faster / render triangles faster than the equivalent sized 4-byte float > map? > > Thomas Goddard wrote: >> Hi Jeff, >> >> Chimera keeps volume data in memory using the same numeric value >> types that are present in the file, for example, 32-bit float, 16-bit >> integer, 8-bit integer. >> >> Volume data is copied only when necessary -- for instance, using the >> Gaussian Filter tool creates a floating point copy of the data since >> Gaussian convolution of integer data produces a floating point result. >> >> Tom >> >> >> Jeff wrote: >>> does chimera store all values as float internally (even if the >>> original map is in 2 byte integers), making the mode of the MRC file >>> irrelevant to chimera's performance? >>> >>> -jeff >>

1 0

Re: [Chimera-users] zone tool and volume rendering
by Thomas Goddard 30 Jan '07

30 Jan '07

Hi Jeff, The Chimera handling of the MRC file origin position is done in Python code chimera/share/VolumeData/mrc/mrc_format.py In summary, if it is an MRC2000 file (ie has xorigin, yorigin, and zorigin header values) then (xorigin, yorigin, zorigin) is used as the xyz coordinates of the volume index (0,0,0) unless xorigin, yorigin and zorigin are all zero in which case the xyz position of the map index (0,0,0) is calculated using integer (ncstart, nrstart, nsstart) header values times the grid spacing (defined by (xlen/mx,ylen/my,zlen/mz)). It is never the case that both (ncstart,nrstart,nsstart) and (xorigin,yorigin,zorigin) are used in positioning the map. Tom Jeff wrote: > > I often use another tool to cut out subregions, that I then load into > chimera - until you told me recently, I did not know I could pre-load > subsampled maps, or be able to keep a 'subsampled overview' present > while selecting various regions > > when using other tools to cut out these smaller areas (either IMOD or > Priism), some tools record displacements, and some don't. the Priism > 'CopyRegion' tool appears to do this by adjusting the 'start pixel' > field in the MRC header. IMOD doesn't do this. the 'copyRegion' tool > also doesn't appear to transfer well to chimera, at least in my test > case the original and subvolume do not register properly. (I don't have > any expectation that it would work properly, but I would adapt if I knew > what chimera expects.) > > I am also not sure what entries chimera uses to keep track of > displacements when it cuts out volumes (I could do that experiment right > now), whether it uses 'start pixel' or 'origin', if 'origin' is recorded > in Angstroms or pixels, etc. Also, I don't know what chimera considers > the origin of a density map - if 'start pixel' is 10,10,10, the map is > 20x20x20 pixels in dim, and the origin is (0,0,0), is the origin at the > center of the map, or at the 0,0,0 pixel coordinate of the map? I have > been assuming that 'origin' has to refer to the map's origin location in > space, but this requires defining where the origin is in the volume map > (I think). if there is a 'start pixel' entry, does this effectively > shift the origin? As an example, if the pixel res is 10 A, will the map > example above now display starting at (10 A, 10 A, 10 A)? Or, is one > (or both) of these entries ignored? Finally, if there is already an > entry in one of these fields, does chimera respect that and just > increment, or does it clobber it with the relative displacement > calculated when excising a subvolume? (this slew of questions is > probably covered by a single answer, but I thought it would be good to > elaborate) > > If I knew, I could more appropriately jump between programs. As I am > writing this, I just got an email from David at Boulder telling me that > IMOD uses the origin entries in the header (but don't know yet what it > considers to be the origin, 'far left' corner or center or something else). > > -Jeff

1 0

Precomputing subsampled map sizes
by Thomas Goddard 30 Jan '07

30 Jan '07

Hi Jeff, Yes, Chimera will complain and not allow precomputed volume subsamples if the size of the subsample grid is not what is expected. In the case you give (1434,1434,110) with subsample grid of size (358,358,26) the subsample grid size is not acceptable. Notice that 4 times the subsample size is (1432, 1432, 104) So a half size bins in the first two dimensions are left off (1434 - 1432 = 2) -- that is fine. But the third dimension is missing 1.5 bins (110 - 104 = 6). The subsample file should have size 27 in the third dimension, ie (358,358,27). I don't know why squeezevol from IMOD dropped off more than a whole bin in the third dimension. Chimera allows fractional bins to be dropped off, but does not allow whole bins to be missing. Tom Jeff wrote: > hi again, > > is there a problem with the precomputed subsamplings needing even > dimensions (I pre-cut my volume to even dimensions so that a binning of > 2 would result in a whole number for the subsampled dimensions, but it > can be hard to do this for a factor of 8 in advance). > > chimera is complaining to me about bad dimensions ('Data foo.mrc of size > (358,358,26) not a valid size for dubsamples of foo.mrc of size > (1434,1434,110). > > is there a program you use to bin / get around this problem? I use > 'squeezevol' from IMOD, I am not sure if it is completely transparent in > its output for what chimera would expect > > -j

1 0

Representation of volume data in memory
by Thomas Goddard 30 Jan '07

30 Jan '07

Hi Jeff, Chimera keeps volume data in memory using the same numeric value types that are present in the file, for example, 32-bit float, 16-bit integer, 8-bit integer. Volume data is copied only when necessary -- for instance, using the Gaussian Filter tool creates a floating point copy of the data since Gaussian convolution of integer data produces a floating point result. Tom Jeff wrote: > does chimera store all values as float internally (even if the original > map is in 2 byte integers), making the mode of the MRC file irrelevant > to chimera's performance? > > -jeff

1 0

Re: [Chimera-users] zone tool and volume rendering
by Thomas Goddard 30 Jan '07

30 Jan '07

Hi Jeff, I'll tell you my experience looking at a tomogram (2048 x 2048 x 76) of an immune synapse for a demonstration I gave last month at a cell tomography workshop. The file (mrc) is too large to conveninently display at full resolution. So I made binned versions 2x and 4x smaller along each axis (file size 8x or 64x smaller). Then I used the precomputed subsamples feature I described before. For some added convenience I then saved the map to a NetCDF file using Chimera because this saves the full matrix and binned versions in a single file (new in Chimera 1.2318). I use the volume dialog data display options to "Adjust step size to show at most 1 Mvoxels", or however many million voxels my computer can handle without getting too slow. Then I use the full view of the data to find regions I'm interested in, then use the subregion selection feature and Crop button to look at a smaller piece and the step size automatically is reduced. If I choose a small region I get full resolution. When doing this you have to keep in mind that once you manually change the step size at the top of the volume dialog it turns off the automatic step size adjustment. So I try to avoid that. Of course I also use the volume planes tool. I think you know all this and probably are using Chimera to the best of its current capabilities. You want to push it to use better compression (like resampling to have better aligned boxes) so you can see more data at full resolution. We don't have that yet. It is not hard to envision much fancier compression implementations that improve performance. Chandra Bajaj at U. Texas is very interested in this and may have software you can use. You might want to try his VolumeRover program. The Chimera EM capabilities were developed for single particle reconstructions. We have done little in Chimera to support work with EM tomograms which are very different in size and content (non-axis aligned structures, membranes, ...). We plan to develop tomogram capabilities and my volume planes tool created last month is a tiny step. But it will take years to implement the obvious stuff we can think of now. Your comments are helpful in guiding the priorities. I don't understand why you are "jotting down displacements" when saving subregions. Chimera puts that info into the MRC header. So why do you need to write it down? Tom Jeff wrote: > thanks Tom, > > with regards to the rotation/resampling, I was thinking mainly in > terms of visualizing large #'s of filaments and membrane, which in > themselves only constitute a small fraction of the actual volume they > traverse. even if re-sampling were done by cubic interpolation onto a > the same dimensioned grid, in some cases the loss in resolution would > be acceptable (definitely would be acceptable for some of my current > needs). i have problems with visualizing my whole maps unbinned but > was thinking that the savings such an approach would represent could > allow me to do that. for instance, this is already a problem for me > with trying to volume render a slab of membrane, which I think might > contain interesting features but I can't get the whole thing to go > unbinned; if it were rotated into the minimal enclosing volume, > interpolation artifacts or otherwise, my feeling was that I might > overcome that (there are also a bunch of filaments attached to the > membrane, and running underneath the membrane, so it is a real > difficulty for me to go back and forth between cutting out subvolumes > and binning them to be manageable, segment out filaments, or membrane, > etc, then try to put all the segmented items back into the same volume > and have a look-see). > > the volume planes tool has proven very helpful in segmenting out the > membranes, though. to get at the filaments that are attached to the > membrane, I was zoning out a region around the membrane (placed a > bunch of markers on the membrane using volume planes tool) about 400 A > in radius, but this proved difficult to render in the entire tomogram > b/c the zoned membrane still encompassed most of the original volume, > etc. so I was back to having to cut regions out of that segmented > membrane map to look further. (thus necessitating me jotting down the > displacements of each subregion, etc.) > > -jeff

1 0

Re: [Chimera-users] using a marker file to zone a volume from python prompt
by Thomas Goddard 29 Jan '07

29 Jan '07

Hi Jeff, I forgot to mention that to run the volumezone.py script without display the Chimera user interface you would invoke it from a shell: % chimera --nogui volumezone.py The --nogui flag tells Chimera not to start any graphical user interface. Tom

1 0

Re: [Chimera-users] zone tool and volume rendering
by Thomas Goddard 29 Jan '07

29 Jan '07

Hi Jeff, You are right that masking a filament that runs along the long diagonal of a box does not reduce the disk space or memory use any. (If you compressed the file then it would make a big difference. But you need to uncompress it to use it so that is of limited value.) So to achieve a space savings you need to resample the volume using a grid aligned with the axes of the filament. The resampling will degrade the data resolution some and it might be reasonable to do it on a grid with half the spacing (making the file 8x bigger). With such a resampled map the rotation and shift could be stored so that it can be placed in the original location in the full map. We don't support that now but when we add resampling we will store those rotation parameters in the map file header (maybe in the "labels" field for an MRC file). The time for Chimera to display a map depends on how large the volume grid is because it scans the entire grid in one pass when using the marching cubes algorithm to compute a contour surface. It also depends on how many triangles make up the contour surface. By eliminating the 99% of a masked file that is zeros using a new orientation and resampling it could compute contour surfaces faster. And the solid (volumetric) rendering should be faster directly proportional to the file size. That may be the more important case for your EM tomograms. I plan on supporting resampling on a rotated grid. But I think in many real uses it will not save much space because you will want to resample with finer grid spacing. But it could be convenient for exporting to analysis or visualization software that is limited to xy-planes. Tom Jeff wrote: > got it, > > what you say makes sense, I can definitely workaround, especially given > that the zone tool does a mask upon saving. > > I have been thinking about the problem I have with filaments (or > surfaces) spanning non-orthogonal regions of a volume, which means that > zoning out a long filament (or surface) that might take a much smaller > amount of space/memory once zoned actually takes up almost the same > amount currently, if I am thinking right. > > as an example, say you have a volume that is 100x100x100. inside the > volume, you have a filament, that takes up 10x10x244 voxels (effectively > spanning from one corner to the mirror/opposite corner of the volume). > in theory, zoning out that filament could take up ~1/40th the space, if > rotated to line up with one of the cartesian axes. I don't know how much > logic it would take to do this; I also don't know how chimera reads > origin information from an MRC file, or how much work it would take to > enable it to read (and use) an euler convention from an MRC file, for > displaying later. This would enable cutting a whole bunch of filaments > out of a volume, and then displaying them all concurrently without > running into the problem of loading in large rectangular prisms of space > that are mainly zeros (if this is currently happening, I don't know). I > guess at that point it makes sense to consider the geometric center of > such a rotated object at its origin, to reduce re-displaying the > object/filament properly in the original orientation to just storing a > shift and the rotation angles (6 parameters). > > the same holds true for membranes - right now, I have an entire membrane > that I segmented with markers, but it essentially runs along a hypotenuse. > > one question that probably negates the above paragraph would be 'does > the masked volume (with mostly zeros) display any faster / take less > resources than an equivalent volume without the zeros'? I have been > assuming this is not the case, that the zoned/masked volume still > entails reading in / accounting for that full rectangular prism, despite > the fact that it is mostly empty / full of zeros. > > thanks for you help, > > -Jeff

1 0

Re: [Chimera-users] using a marker file to zone a volume from python prompt
by Thomas Goddard 29 Jan '07

29 Jan '07

Hi Jeff, I've attached a script that opens a density map and a Chimera marker file and writes a new volume corresponding to a zone around the markers. It does not require showing the Chimera user interface to do this. Perhaps you don't need this now (I just got your follow-up email.) but perhaps you will find it useful in writing other scripts. Karin Gross at Max Planck Institute of Biochemistry has written a Chimera extension to slice tomograms at arbitrary angles. I saw it when I visited Munich in December. They also were planning on allowing marker placement on those planes. The volume planes tool could be made to average neighboring planes. Currently it simply displays a NxMx1 subvolume in solid rendering mode. Since there is no existing code in Chimera to support averaging adjacent planes that would have to be added. Probably the easiest solution is to add the adjacent planes in Python and create a new single plane volume on the fly. Tom Jeff wrote: > thanks Tom, > > I will try that. alongside that, is there a way for me to run a 'zone' > function and save a subregion from a python command line (no chimera > launched, just import a library?)? > > I have used the volume planes tool to generate a marker set of a > membrane, it was handy. But, it only handles orthogonal directions, it > can't provide an arbitrary interpolant (which is a large reason I rely > heavily on IMOD's slicer tool); this functionality would be very useful. > A second addition to the volume planes tool would simply be allowing > averaging several slices together (reduces noise) in the direction > normal to the planes, which may already be there (the documentation > online said something about this feature, but it wasn't clear to me) > > it the volume planes tool were able to handle an arbitrary > view/interpolant, then I wouldn't have to depend on the slicer tool, and > the previous request about rotating markers that was related to that > requirement (different features available in different slicer > orientations in IMOD, necessitating a rotation prior to visualization > with the volume planes tool in chimera, thus requiring old marker set > from original volume to be rotated to be used in new volume, etc) would > effectively be voided > > > -Jeff # ---------------------------------------------------------------------------- # Save a volume zone around a given set of markers as a new volume file. # def save_zone(volume_path, markers_path, radius, volume_out_path): # Read zone points atoms, bonds = marker_set_atoms_and_bonds(markers_path) from chimera import Xform xform = Xform() # Identity transform for marker points from SurfaceZone import path_points points = path_points(atoms, bonds, xform) from VolumeData import open_file_type vol = open_file_type(volume_path, 'mrc') # Get containing volume subregion ijk_min, ijk_max = containing_subregion(vol, points, radius) from VolumeData import Grid_Subregion subvol = Grid_Subregion(vol, ijk_min, ijk_max) # Make new volume with zeros outside radius from VolumeViewer import zone_masked_grid_data masked_vol = zone_masked_grid_data(subvol, points, radius) # Write new volume from VolumeData.mrc import write_mrc2000_grid_data write_mrc2000_grid_data(masked_vol, volume_out_path) # ---------------------------------------------------------------------------- # def marker_set_atoms_and_bonds(markers_path): from VolumePath.markerset import load_marker_set_xml msets = load_marker_set_xml(markers_path) atoms = [] bonds = [] for mset in msets: atoms.extend(map(lambda m: m.atom, mset.markers())) bonds.extend(map(lambda l: l.bond, mset.links())) return atoms, bonds # ---------------------------------------------------------------------------- # def containing_subregion(vol, points, radius): from VolumeViewer import points_ijk_bounds, clamp_region ijk_min, ijk_max = points_ijk_bounds(points, radius, vol) ijk_min, ijk_max = clamp_region((ijk_min, ijk_max, None), vol.size)[:2] from math import floor, ceil ijk_min = map(lambda x: int(floor(x)), ijk_min) ijk_max = map(lambda x: int(ceil(x)), ijk_max) return ijk_min, ijk_max # ---------------------------------------------------------------------------- # volume_path = '/usr/local/src/chimera-demos/volume/examples/tricorn_map.mrc' markers_path = '/usr/local/src/staff/goddard/chimera/devel/ZoneScript/test.cmm' radius = 5 volume_out_path = '/usr/local/src/staff/goddard/chimera/devel/ZoneScript/test.mrc' save_zone(volume_path, markers_path, radius, volume_out_path)

1 0