Each of these headings will probably get their own page eventually...
- Shell scripts: useful bits and pieces for bash scripting.
- Version Control (old but useful): using SVN to keep track of code.
- Version Control (newer and better): using Mercurial to keep track of code/docs (links to new page).
- Imagemagick: How to use the command-line tool 'convert'.
- Useful Unix commands: random useful commands I keep forgetting how to call.
- LaTeX: a few hints for LaTeX documents.
- Gnuplot
- Mac OS X stuff
- Visualisation of data with VisIt
Mac OSX collection of useful things
- Convert PDF presentations to Keynote slides. This worked really well the only time I have used it so far. Download it from http://www.cs.hmc.edu/~oneill/freesoftware/pdftokeynote.html; the website explains it pretty well.
From the description: PDF to Keynote converts PDF-based presentations (e.g., made using LaTeX with Prosper, Beamer or foiltex) into Apple's Keynote format. Once translated into Keynote format, every page of your original PDF file becomes a PDF image, properly scaled and centered on its own slide. Although intended for presentations, you can also use it to create slides for the pages of regular documents. Because each page of your original document becomes a PDF image, you can't easily edit those pages except by adding new content over the top to mask things out.- Lock Screen: For some reason there is no built-in and switched-on GUI menu option to lock the screen on a MacBook (for OSX 10.6 at least). But there is a way to do it, which I found from d'internet here: http://osxdaily.com/2011/02/10/lock-mac-desktop-via-menu/
The option is there in the keychain access utility, but it is not shown on screen by default, so you just need to enable it in Preferences - ''Show Status in Menu Bar''. The same website also points out that the keystroke combination Control-Shift-Eject will also lock the screen. So you don't really need the GUI option, but I went 3 years using Macs without knowing this...
Using VisIt for data visualisation
VisIt is a really powerful piece of software developed at LLNL for looking at large datasets, but it is not trivial to install, or to get it to read your data. There is lots of information on the LLNL VisIt website, from the visit-users mailing list, and on the visitusers.org wiki.
Installing: You can download the binaries, and the Mac one at least works well. But on Linux they are often not very portable, and I have found that I get a more stable installation when I compile from source. First you need to make sure you have the right libraries installed on the system; graphics acceleration development libraries are important (e.g. Mesa/OpenGL dev. headers on linux). Also, each release has its own idiosyncracies, which don't generalise to all systems, so sometimes things need to be hacked a little.
You can try installing with this script which I wrote. It downloads a script from LLNL to build the version of VisIt that you requested, compiles the source code in a user-specified directory, and then installs to another user-specified path. If it bugs out, you need to look in the build_visitX_Y_Z_log file, where X_Y_Z is the version of VisIt you tried to install (2.6.1 is the latest). This file should be in the compilation directory. This script compiles the serial version of VisIt.
VisIt can also be compiled with parallel support, so that it can use multiple cpus for reading data and generating images. This can be faster for visualising large datasets. I have a compile script with parallel support enabled here; it is very similar to the serial script except that the argument --parallel is passed to the build_visit script.
Note that I can't guarantee anything about the scripts, not even that I will help if it doesn't work (if I have time I will help). If the script fails, it is probably to do with a mismatch between your linux/unix system and the systems which VisIt is tested on, so your best bet is to write to the visit-users mailing list for help.
- Generating movies and figure from scripts: This is very useful, especially in case you need to alter a figure slightly, or want to produce a B+W version. If you generated the figures using the graphical interface, it can be difficult to recreate the same figure again (unless you saved the session). I generally set VisIt to record how I set up the figure using the graphical interface, copy the output into a python script, and then edit the script for fine-tuning the figures. It means I can be sure that I can easily regenerate the same figure in 2 years time, if needed. I'll put an example script here soon...
LaTeX
- \mathbf{} makes normal bold-face Roman characters, and doesn't allow italics by default. So to get bold italic math characters you can include the following (from here):
\DeclareMathAlphabet{\mathitbf}{OML}{cmm}{b}{it}- Need the LaTeX name for a symbol, but can't remember it? Draw it here detexify.kirelabs.org/classify.html and get the LaTeX code!
- Need many sub-figures as part of a single large figure, on multiple pages? Use the subfig class and \ContinuedFloat e.g.
% HEADER \usepackage{graphicx} \usepackage{subfig} ... % DOCUMENT \begin{figure}[h] \centering \includegraphics[width=0.9\textwidth]{fig1a.eps} \includegraphics[width=0.9\textwidth]{fig1b.eps} \caption{this is the caption. \label{fig:fig1}} \end{figure} \begin{figure} \ContinuedFloat \centering \includegraphics[width=0.9\textwidth]{fig1c.eps} \includegraphics[width=0.9\textwidth]{fig1d.eps} \includegraphics[width=0.9\textwidth]{fig1e.eps} \caption{This is the caption (again).} \end{figure}
Version Control (SVN)
Subversion is the version control system I am using. I used to use CVS, but I found SVN to be better so I switched. This page is mostly to remind myself of how to do things with SVN because I have only used most of the commands a handful of times.Update: I now also use Mercurial, which seems basically the same as SVN, but a bit better for some things such as merging and conflict resolution (and it is distributed, not centralised). Some info about Mercurial will make its way onto this page eventually, once I start using it more. See mercurial.html for more details.
I have a few environmental variables set in my bashrc file for SVN:
export SVN_EDITOR=/usr/bin/vim export SVN_SSH="ssh -l[username] -p[ssh-port]"Creating a project
First create a directory and a file or two as a skeleton for the project.
Then create a SVN "repository" for the files.
Then import the directory and file(s) into the SVN repository.jm@localhost:~$ mkdir temp/; mkdir temp/new_project jm@localhost:~$ touch temp/new_project/log.txt jm@localhost:~$ svnadmin create /path/to/archive/NewProject jm@localhost:~$ svn import ~/temp/new_project file:///path/to/archive/NewProject/ -m "initial import" Adding ~/temp/new_project/log.txtChecking out a project
Now the project can be checked out remotely using the first line below, or when on the local machine using the second line below.
jm@localhost:~$ svn co svn+ssh://host.university.country/path/to/archive/NewProject ./new_project/ jm@localhost:~$ svn co file:///path/to/archive/NewProject ./new_project/Commiting changes
...Checking information on files/directories
...Merging a branch back to the trunk
I got the information for this from the SVN book and this blog.
You first need to have an up-to-date copy of the trunk. Then find the revision when the branch was created:
jm@localhost:/$ svn log --stop-on-copy svn+ssh://U.R.L/path/code/branches/RTupdate | tail -n5 ------------------------------------------------------------------------ r359 | jm | 2011-10-06 12:29:45 +0200 (Thu, 06 Oct 2011) | 1 line Branch for updating the raytracing module. ------------------------------------------------------------------------Change directory to the trunk dir and see what the current revision is and make sure it is updated.
jm@localhost:/path/to/projects/code/trunk$ svn update At revision 396.Now merge the branch to the trunk.jm@localhost:/path/to/projects/code/trunk$ svn merge -r 359:HEAD svn+ssh://U.R.L/path/code/branches/RTupdate . Conflict discovered in 'extra_libraries/install_all_libs.sh'. Select: (p) postpone, (df) diff-full, (e) edit, (mc) mine-conflict, (tc) theirs-conflict, (s) show all options: s (e) edit - change merged file in an editor (df) diff-full - show all changes made to merged file (r) resolved - accept merged version of file (dc) display-conflict - show all conflicts (ignoring merged version) (mc) mine-conflict - accept my version for all conflicts (same) (tc) theirs-conflict - accept their version for all conflicts (same) (mf) mine-full - accept my version of entire file (even non-conflicts) (tf) theirs-full - accept their version of entire file (same) (p) postpone - mark the conflict to be resolved later (l) launch - launch external tool to resolve conflict (s) show all - show this list Select: (p) postpone, (df) diff-full, (e) edit, (mc) mine-conflict, (tc) theirs-conflict, (s) show all options: dc @@ export CXX=g++ export FC=gfortran <<<<<<< MINE (select with 'mc') (13,192) ... MY (TRUNK) COPY OF FILE ... ||||||| ORIGINAL (11,0) ======= >>>>>>> THEIRS (select with 'tc') (11,2) Select: (p) postpone, (df) diff-full, (e) edit, (r) resolved, (mc) mine-conflict, (tc) theirs-conflict, (s) show all options: mf --- Merging r360 through r396 into '.': U extra_libraries/install_all_libs.shWhen there are conflicts, the branch version is denoted 'theirs', the current directory (trunk) is denoted 'mine', so you can use 'mc' to use trunk versions for conflicting regions (or 'mf' to use the full trunk file), or equivalently 'tc' and 'tf' to have branch code replace trunk code. You can display conflicts with 'dc', of the diffs with 'df', but I find the diffs are less useful because you are comparing three files and 'dc' clearly states 'MINE', 'THEIRS', or 'ORIGINAL'. In any event, I found it simpler to go to the trunk and branch copies of the source files and compare them myself and then decide which one to use.
At the end of it all I got:
U bin_serial/Makefile.serial.icgenerator C bin_serial/compile_phalanx.sh D bin_serial/aa_compile.sh --- Merging r360 through r396 into 'ics': U ics/get_sim_info.cc U ics/icgen.cc U ics/icgen.h U ics/blast_wave.cc Summary of conflicts: Tree conflicts: 3 jm@localhost:/path/to/projects/code/trunk$The conflicts were from files that were deleted on the trunk since the branch was created, and I guess were not deleted on the branch. You can now see what files have changed with "svn status", and compare them against the branch files (one-by-one) with diff. As expected, all of the trunk files were identical to the branch ones except where there were conflicts and I kept the trunk copy.
So the sequence of commands is:
svn log --stop-on-copy ...to see what revision the branch started. svn updatein trunk directory to make sure it is up-to-date. svn merge ...to merge the branch to the trunk.- Probably lots of conflict resolution.
- Checking that everything is correct
svn committhe trunk directory with a comment to say it is merged from the branch (with revision numbers on the branch, since it may continue to be used).- Delete the branch if no longer using it.
Imagemagick
Imagemagick is a very useful tool for converting figures between formats, cropping borders, shrinking images, etc., and it has a command-line interface so you can put it in scripts. Here are some examples I find useful (most of them taken from the imagemagick website examples).
- Convert eps to jpeg (I think density is dots-per-inch, and quality is the jpeg quality flag; density is most important for figures):
$ convert -density 300 -quality 100 file.eps file.jpeg $ convert -geometry 568x256 -density 300 -quality 100 image.eps image.jpegThe geometry tag sets the size of the output image, and I think it's a good idea to set it to be a multiple or fraction of the eps bounding box (from `head image.eps | grep BoundingBox'). The default (I think) is to set them to be the same.- Crop an image: the first two numbers are the number of pixels in the output, and the last two are the x and y offsets from the top left corner to start the cropping from. Putting an exclamation mark at the end of the geometry item resets the new image origin to the top left corner of the new image (otherwise it stays relative to the old one!).
$ convert -crop 955x325+175+225 input_file.tif output_file.tif $ convert -crop 955x325+175+225! input_file.tif output_file.tif- Annotate an image: pointsize refers to the size of the text; annotate gives the offsets from the top left (of the original image if you cropped it!) to start writing, and the text is in single quotes.
$ convert input_file.tif -pointsize 20 -annotate +250+275 \ 'this is an annotation' output_file.tif- Make an image smaller (by resampling):
convert terminal.gif -resize 50% half_terminal.gif- Draw a line with a certain thickness (coordinates are from the top left in pixels, and refer to the start and end of the line).
convert source_file.gif -stroke white -strokewidth 3 -draw 'line 20,20 20,80' output_file.gif- Draw two arrows: The following lines generate two coordinate axes, with X as horizontal and Z as vertical. The background is white and the foreground is black, although they can of course be changed. The idea is to draw a line, put an arrowhead on the end of it, and position an axis label off the end of each arrow. Positions are measured from the top left by Imagemagick, and positive rotations are in degrees from the X-axis. The triad is then merged onto a figure twice (i.e. in two places) using the "composite" command, which I think is another command-line tool from ImageMagick. It's not a very stylish figure, but it shows what can be done if required.
arrow_head="path 'M 0,0 l -15,-5 +5,+5 -5,+5 +15,-5 z'" convert -size 275x275 xc:white temp.gif convert -size 800x800 xc:green Background.eps convert temp.gif -stroke black -strokewidth 10 -draw 'line 30,245 30,075' temp.gif convert temp.gif -draw "stroke black fill black translate 30,065 rotate -90 scale 3.5,3.5 $arrow_head " temp.gif convert temp.gif -pointsize 72 -font "Times-Roman" -fill black -annotate +9+63 "Z" temp.gif convert temp.gif -stroke black -strokewidth 10 -draw 'line 25,245 200,245' temp.gif convert temp.gif -draw "stroke black fill black translate 210,245 rotate 0 scale 3.5,3.5 $arrow_head " temp.gif convert temp.gif -pointsize 72 -font "Times-Roman" -fill black -annotate +215+270 "X" temp.gif convert temp.gif -resize 100x100 eps3:triadXZ_f2.eps mv temp.gif axes_triad_XpZp.gif composite -geometry +10+575 triadXZ_f2.eps Background.eps temp_fig2.eps composite -geometry +10+263 triadXZ_f2.eps temp_fig2.eps temp2_fig2.eps
Unix Commands
Concatenate pdf files (found here):gs -q -sPAPERSIZE=a4 -dNOPAUSE -dBATCH -sDEVICE=pdfwrite -sOutputFile=output.pdf file1.pdf file2.pdf [...] lastfile.pdfDelete files with zero size (found here):find -maxdepth 1 -type f -size 0 -print0 | xargs -0 rm -fDelete files matching a pattern (e.g. file extension) (found here):find . -type f -name "FILE-TO-FIND" -exec rm -f {} \;
Shell scripts
- Deleting text between two identifiers in a file (found here), which can be useful for generating clean source code without all of the debugging ifdefs.
sed '/\#ifdef DEBUGGING/,/\#endif/d' file.cpp > output.cppThis can of course get confused if you have nested ifdefs...- Replace text in a variable: If FILE=hello.png is a PNG image, and you want to convert to an image called hello.eps, then you can do this:
IMGFILE=`echo $FILE | sed 's/png/eps/'` convert $FILE eps3:$IMGFILE- Fixed number of digits in a filename: If you are generating a sequence of files, and want them numbered with a counter with exactly N digits, you can use this:
$ BASE=myfilename $ for i in `seq 0 10`; do > FILE=`printf "%s_%03d.tif" $BASE $i`; echo $FILE > done myfilename_000.tif myfilename_001.tif myfilename_002.tif myfilename_003.tif myfilename_004.tif myfilename_005.tif myfilename_006.tif myfilename_007.tif myfilename_008.tif myfilename_009.tif myfilename_010.tif- Multiplying floating point numbers: unfortunately bash and other shell scripts can only do integer arithmetic, so for floating point operations you need to use another tool, such as 'bc'. The following line divides 10 by 3.0 using 'bc'; the 'scale=2' command indicates the calculation should be performed to at least two decimal places.
$ ii=10; FLT=$(echo "scale=2; $ii/3.0" | bc); echo $FLT 3.33
Gnuplot
- Solar mass symbol in Gnuplot (adapted from this discussion):
gnuplot> set terminal postscript enhanced color eps font "Times,16" \ fontfile "/usr/share/texmf-texlive/fonts/type1/public/amsfonts/cm/cmsy10.pfb" gnuplot> set label 1 "{/CMSY10 \014}" at 0,0 gnuplot> set out "test.eps" gnuplot> plot x gnuplot> set out- Symbols: 150 degrees (angle) is
150{/Symbol \260}- Symbols: The Greek symbol theta is
{/Symbol q}