Preliminaries/Setup
Environment variables that need to be set:
export DIFMAP_LOGIN=/home/mlister/.www/scripts/macros
export PGPLOT_DIR=/usr/local/pgplot
export LD_LIBRARY_PATH=/usr/local/pgplot
export PGPLOT_DEV=/xserve
export SCHED=/home/mlister/sched
export GROUP=astrophys
export PATH=./:/usr/local/bin:/bin:/usr/bin:/usr/X11R6/bin:/home/mlister/.www/scripts/:/project/miranda/mlister/perl/:/sbin
export PRINTER=knuth
export EDITOR=gedit
export TEXINPUTS=.:/usr/share/texmf/tex//:/project/miranda/mlister/aastex52//:/usr/share/lib/texmf/tex//:/opt/local/share/teTeX/texmf/
export PERL5LIB=.:/usr/lib/perl/5.8.4:/home/mlister/.www/scripts:/project/miranda/mlister/perl/scripts:/project/miranda/mlister/perl
export WWW=/project/astro/www/MOJAVE/
umask 002
If you are using bash shell, these need to go in your .bashrc or .profile (whatever
gets read automatically upon login). To find out what shell you are using, type
echo $0
To change your default shell (to /bin/csh for example), log in to bohr.physics.purdue.edu and use the 'chsh' command. The change might take up to an hour to propagate through the system.
You also should be part of the 'astrophys' group. (Type groups to find out). If not,
let me know asap.
Computers/Disks
Our group has four machines that can run AIPS:
oberon: rm 391: Intel XeonE5530 4x2.40GHz Dell Precision T5500 4Gb (Jul 2009)
ophelia: rm 326 : Intel Xeon E5345 4x2.33GHz Dell Precision 690 4Gb (Apr 2007)
miranda: rm 370A: Intel Xeon E52643 4x3.3GHz Dell precision T7600 8 Gb (June 2013)
titania: rm 309: Intel Xeon E5620 4x2.40GHz Dell Precision T5500 16Gb (June 2010)
prospero: (retired) Intel Xeon E5345 4x2.33GHz Dell Precision 490 4Gb (Apr 2007)
umbriel: (retired)
ariel: (retired)
Don't run any large CPU tasks on miranda without asking me first.
Each machine has a /project directory (/project/titania, etc. ) that can be
used to store data files. cd to the directory and type
df -h
to show available disk space. If you don't have a subdirectory of your
own, create one, or ask me to do it for you.
Most programs except AIPS will run remotely from any other linux machine on the internet, or a windows machine with the right emulator software (note I have no experience with the latter). For AIPS, it's a little trickier getting the TV window to appear remotely, and it's usually painfully slow unless you're on the dept 1 Gb/s line. You have to use 'aips tv=displaymachine:servermachine' . Note that each machine can
only host one TV server at a time. This means that you could potentially 'steal' the TV from the person sitting at the host computer. Using 'aips remote' will give you a text only AIPS session, but is very fast.
The MySQL Database
------
The group has a mySQL database that is the primary storage area for derived quantities associated with
blazars in the MOJAVE program. To access it, type
mysql -p -u agn -h galaxies galaxies
(See Prof. Lister for the password)
The database 'galaxies' contains various tables. To list them,type
mysql> show tables;
Information on individual epochs is stored in the 'epochs' table.
To show the column (field) names in the epochs table:
mysql> show columns from epochs;
or
mysql> describe full columns from epochs;
SQL has a rather complex query language, but is well documented at mysql.org
Every query must end with a semi-colon.
Examples of simple queries are: (% is a wildcard, * means select all fields)
mysql> select * from epochs where obscode like 'bl137%';
mysql> select source,z from sources where MOJAVE_1='Y' order by source;
(selects sourcename and redshift for all 135 original MOJAVE sources)
Here's an example of a complicated one:
mysql> select source,z,lumdist,maxMu,max2cmvlbaflux,format(maxMu*lumdist/63.24217/(1+z)/1000,2) as beta_app,format(20.07786+log10(lumdist*lumdist/(1+z)*max2cmvlbaflux),3) as logL,maxMu_ref from sources where MOJAVE_1='Y' and spect_class != 'GPS1' and opt_class !='G' order by decsign,logL;
Joining information from different tables can be a bit of an art form: e.g.,
mysql> select z,muasyr,dmuasyr from components join sources where sources.source=components.source and sources.MOJAVE_1='Y' and sources.fitstatus like 'finish%' and components.muasyr > components.dmuasyr and sources.z > 0 order by sources.z;
Although the database is backed up nightly, be extremely careful with the 'update' command, as you can easily wipe out thousands of records at once (and there's no 'undo' function!). Same goes for the 'delete' command. Since the default is 'perform operation on all records', both of these commands should never be run without a 'where' qualifier, e.g.:
mysql> update epochs set needs_recalib='N' where epoch='2005-09-05' and source='2201+315' ;
Loading data from disk
To obtain data from NRAO, go to archive.nrao.edu and use the advanced query tool
to locate the right experiment. Follow the instructions and download the data
to a /project directory with sufficient disk space. If you see a large single file with
your expt code and frequency, download it, otherwise, you'll have to download the
individual segments. Be careful, sometimes there are two versions of the exact same segment listed, with the same time range. Pick the one with the more recent file modification date. If the experiment consists of multiple segment files, check the time listing carefully, and select only the ones you need.
Once you receive the email from NRAO, run the makeSCRIPT script in ~mlister/.www/scripts, after copying the /home/mlister/a/data/vlba_gains.keyfile to your data directory
Follow the instructions in the <expcode>.SCRIPT file that is created in the <expcode> subdirectory.
Note that in the script inputs, the filenames are preceded by 'U', which is the linux environment variable for where your VLBA data file resides. If you don't specify the U in DATAIN, AIPS will assume that you are loading
from the default FITS directory, which is /project/miranda/AIPS/FITS
It is advisable to avoid using this directory. An annoying feature of AIPS is that once you start aips, it is too late to set any new environment variables like U. You must exit
AIPS with
AIPS>kleenex
, set the new variables, then start aips again.
Note that the DATAIN variable has a maximum of 48 characters, including the path
and filename. Leaving off the trailing quote in DATAIN 'U:fits means AIPS will preserve the lowercase, otherwise it will assume you are loading FITS1, FITS2, etc. For more info within AIPS, type help DATAIN
Processing non-MOJAVE VLBA data
------
Once the datafile is loaded, the procedure to follow is outlined in
The main steps in processing VLBA data are flagging bad data, calibrating the amplitudes of the visibilities, and calibrating the phases of the visiblities. Here are
more details:
Making sure your file has the correct tables attached. Set indisk to the disk
which you loaded your file to, then type ucat. Then type getn X , where X is the file number of your file. Type imhead to view details of the file and its tables. The crucial tables are:
TY: system temperature table.
GC: antenna gain table
FG: flag table
PC: pulse calibration table
NX: index (scan listing) table
WX: weather table (only needed for experiments >= 15 GHz)
If your experiment was observed after the year 2000, it will likely have all these files correctly attached. (Note: there have been problems with PC tables on some files downloaded from the archive, you'll have to delete it and recreate them with PCLOD). For experiments before 2000, or those that include non-VLBA antennas, extra work is required to get correct tables. Details are in the AIPS cookbook.
For experiments below 8 GHz, a correction for ionospheric effects should be considered.
For experiments involving phase-referencing (a special technique used to image very weak sources), corrections need to be made to the earth orientation parameters.
Bad data need to be flagged (removed).
If your experiment has 2-bit sampling , run ACCOR . To find the sampling rate, go to, e.g
change the apr06 as appropriate (an experiment observed in September 1996will be sep96). Find your experiment code and click on it. The .sum file lists all the gory details. None of the BL111, BL123, or BL137 experiments have 2-bit sampling, but the BL149 ones after 2008-09-12 do.
For experiments >= 15 GHz, you will need to enter the expected flux density of each source using SETJY.
Run APCAL to establish the absolute flux density scaling for all antennas. For experiments >= 15 GHz, you will need to set opcode='OPAC'.
Flag the resulting solution (SN) table using SNEDT and apply with CLCAL
If your experiment has full polarization or uses phase-referencing, you must run VLBAPANG to correct the phases for parallactic angle rotation.
Apply the pulse-cal phase corrections using VLBAPCOR.
Fringe fitting is an optional step - if the pulse cals are good, and the source positions are well known, you may skip it entirely. The pulse cals also usually do a good job calibrating the cross-hand phases. For 43 GHz and above, the pulse cals cannot be used. They also cannot be used for the 512 Mbps MOJAVE expts.
If your experiment has full polarization you may have to run CRSFRING to calibrate the cross-hand phases. (see step 13 of the MOJAVE data reduction below)
Calibrate the bandpass with BPASS
Apply all calibrations with SPLIT
Run a point-source phase-only selfcalibration with CALIB on the splitted source files.
Output all the self-cal'd files to linux disk with FITTP. See the DIFMAP section below.
Steps for reducing MOJAVE VLBA data
------
Because all MOJAVE experiments are identical, except for the observing frequency and sources that are observed, a standard script can be used to reduce them. Only a few modifications to the script are usually necessary.
If you are reducing MOJAVE data for the first time, start aips, enter your user number (NNN) and then
AIPS> ehex NNN
to see what your user number is in extended hex form. If it returns less than 3 digits,
prefix the extended hex value with a zero, e.g. 1025 = 0SH in extended hex.
then type
linux>cp ~aips/RUN/MYPROC.0SH ~aips/RUN/MYPROC.EEE
where EEE is your extended hex user number.
Setting up linux directories
cd to a /project directory with at least 4 Gb disk space. The steps below assume you are reducing an experiment BL137G:
linux> makeSCRIPT.pl BL137G
linux> cd BL137G
linux> setenv U $PWD (csh shell)
Load the BL137G.SCRIPT file in a text editor, cd to your BL137G directory, then copy and paste the commands under step 1 of the script into the linux prompt.
2. Loading the data
Obtain the data file from archive.nrao.edu and store it in your BL137G directory using the makeSCRIPT script.
Cut and paste the commands in the <expcode>.SCRIPT file under step 2 into the aips prompt.
Note: if the dataset uses 2 bit sampling, you must run ACCOR. (see the AIPS cookbook)
3. Flagging and choosing reference antenna
After recreating the NX and CL tables by cutting and pasting the first few lines of step 3 into AIPS, you will need to flag data based on high system temperature values with EDITA. Look for abnormally high values, and flag in all stokes/IFs. To pick a menu option, highlight with the cursor, press 'A'. Check the AIPS message window for info.
To select a reference antenna, generally pick FD unless it is either missing, has many bad (flagged) times, or has very poor weather. In that case, refer to the TSYS.PS plot, and the weather files
under
(change apr06 to the mmm/yy of your experiment). The weather plots are, e.g.
bl137ctsm.LA.ps.gz
A good reference antenna other than FD is one of PT, LA , KP or NL, and experiences no rain or snow, has less than 10 minutes downtime, no electronic/mechanical failures, and has no abnormally high system temperatures in TSYS.PS. You should go through the antennas in the order above until you find a suitable reference antenna.
Once you have selected a reference antenna, do a global search and replace
on antnum('FD') in the script file where 'FD' is your reference antenna code.
If you are reducing a 8 or 12 GHz experiment, do step 5, otherwise, do step 4.
4. (15 GHz only) A-priori amplitude correction with opacity correction.
Paste the text from step 4 into AIPS
5. (Below 15 Ghz) A-priori amplitude correction with no opacity correction.
Paste the text from step 5 into AIPS
6. Paste the steps from step 6 into AIPS. Edit the SN table in the same
manner as you did with EDITA.
7. Run CLCAL by pasting the text from step 7 into AIPS. Temporarily exit out of aips
with ctrl-z and type gv $U/APCAL.PS to inspect the output SN table. Return
to AIPS by typing fg at the linux prompt.
8. Paste the text from step 8 to perform the parallactic angle correction.
9. For this step you need to identify a source in the experiment that is very bright, and point-like. Check the flux densities and images of the sources on the MOJAVE pages.
Once you've identified the source, hit ctrl-z to return to linux, then type
grep 2134+004 VLBASUMM.TXT (where 2134+004 is the source you've picked)
This will list a series of scan numbers and times that contain this source. Check
to make sure the time of the middle scan in the list doesn't coincide with any time
period you've previously flagged out for an antenna because of weather/electronics.
If so, pick another scan near the middle of the list and record its scan number. Go
back to AIPS with 'fg' and edit the script text under step 9 to change TIMER=scantime(75) to the scan number you've selected. Copy the text of step 9
to the AIPS prompt. Check on the TV that it plots 9 baselines, and the data look
ok on all baselines.
10. Copy the text from step 10 into the AIPS prompt.
11a. Calibrating phases across frequency band
Copy the text from step 11 into the AIPS prompt. If you get an error message
here, it may be because the PC table is missing some info for some antenna, or the scan time you selected in step 10 doesn't contain data for all baselines.
11b. Global fringe fit
If all the sources are point-like and above 200 mJy, you may wish to run step 12. Otherwise, you can skip this step entirely. If you weren't able to use the pulse-cals, then this step must be done.
12. (8 Ghz only) If you are reducing an 8 Ghz MOJAVE experiment, you will have to split the data at this point into 8.1 and 8.4 Ghz files, and run all the following steps on each new file.
13. Crosshand phase adjustment
For this step, the source in your reference scan time (scantime) must be polarized. Check the MOJAVE page for at least 30 mJy of polarized flux). If not, you need to choose a different scantime. Paste the text from step 13 into the AIPS prompt. Check the output in the message window and make sure there are no failed solutions. If so, you
need to delete the SN and CL tables that were created by this task, pick a different scantime, and rerun step 13. Check the POSSM_CROSS.PS plot to see if most
of the phases are flat.
14. Bandpass calibration
For this step, use the same scantime as you used in step 9.
15. Save tables and apply calibration
At this point save your calibration tables with the text in step 15, SPLIT the source
files from the main one, do a point source selfcalibration on them, and output to $U.
Make sure the filenames are correct, in that they use the correct frequency code (see the script).
At this point, you are ready for DIFMAP. Take a well deserved break.
Self-calibrating and Imaging in DIFMAP
DIFMAP is a very useful program written by Martin Shepherd of Caltech for processing and imaging interferometer data. Although all the same functionality exists in AIPS,
DIFMAP is far faster and simpler to use.
To install DIFMAP, download the source code from the Caltech site; you will first need to install
pgplot (using apt-get, or compiling it yourself), and g++, fort77, libX11-dev, libncurses-dev.
To use DIFMAP, make sure you have your environment variables set up as described on page 1, then cd to the directory containing your data, and type
linux>difmap
To load an interferometer data (uv) file, type
difmap> obs <filename>
If you're doing a MOJAVE epoch, you will want to load the uvf_raw_edt file.
At any point if you wish to temporarily exit DIFMAP to go back to the linux command prompt, hit <ctrl-z>. Typing 'fg' at the command prompt will return you to where you left off in DIFMAP.
To load a file that you've previously saved with 'save <filename>' in DIFMAP, type
difmap>@<filename>.par
0. Setting the map size and selecting stokes I: this can be done by typing 'go' at the difmap prompt.