Skip to content

Adding a New Epoch

VAST Tools contains packaged observing information and MOC files for each epoch of the VAST Pilot Survey that is available. When a new epoch is ready to be added to VAST Tools the following must be completed.

Generating the VAST Tools metadata

Epoch information csv files

These files contain the observing information of each individual beam that makes up the epoch The example below shows the expected format.

vast_epoch09_info.csv

SBID,FIELD_NAME,BEAM,RA_HMS,DEC_DMS,DATEOBS,DATEEND,NINT,BMAJ,BMIN,BPA
11248,VAST_1724-31A,0,17:22:15.223,-30:51:39.82,2020-01-12 02:52:19.947,2020-01-12 03:04:16.583,73,15.970375,12.021243,-42.804366
11248,VAST_1724-31A,1,17:27:08.777,-30:51:39.82,2020-01-12 02:52:19.947,2020-01-12 03:04:16.583,73,15.916086,11.722993,-41.766406
11248,VAST_1724-31A,2,17:22:13.572,-31:54:39.6,2020-01-12 02:52:19.947,2020-01-12 03:04:16.583,73,15.960169,12.041986,-42.239253
11248,VAST_1724-31A,3,17:27:10.428,-31:54:39.6,2020-01-12 02:52:19.947,2020-01-12 03:04:16.583,73,16.000031,12.015099,-42.539542
The description of these columns can be found here.

These files use data from the ASKAP_SURVEYS repository. These files can be automatically generated by pointing vasttools.tools.create_fields_csv to a local version of ASKAP_SURVEYS. See the function documentation for details.

MOCs and STMOCs

Information about the survey coverage is stored as multi-order coverage maps (MOCs) and space-time MOCs (STMOCs) in the vasttools/data/mocs/ direcory. The vasttools.tools.gen_mocs_epoch function can be used to generate all MOC data for a single epoch. This function will generate an individual MOC for each pointing, a MOC containing the combined footprint of all pointings in the epoch, and an updated STMOC describing the coverage of the full survey to-date. Currently only the latter two are required to add a new epoch to the package, as individual field footprints do not vary from epoch to epoch and are stored in vasttools/data/mocs/COMBINED and vasttools/data/mocs/TILES.

However, adding observations with different footprints (e.g. the Phase II survey high frequency observations) will require new footprint MOCs to be created. This will also require new field MOCs (e.g. vasttools/data/mocs/VAST_PILOT_FIELD_1.fits) to be generated, although no automated process for this exists yet.

Epoch Addition Steps

To add a new epoch, users should take the below actions.

  • Ensure that you have access to the necessary data, i.e. the ASKAP_SURVEYS repo and all images from the epoch and image type of interest.

  • Generate the epoch metadata using the methods described above.

  • Place the epoch information csv file in vasttools/data/csvs/, named vast_epochXX_info.csv, where XX is the two digit zero padded epoch number.

  • Place the field SkyCoord pickle file in vasttools/data/pickles/, named vast_epochXX_fields_sc.pickle, where XX is the two digit zero padded epoch number.

  • Place the full-epoch MOC in vasttools/data/mocs/, named VAST_PILOT_EPOCHXX.moc.fits, where XX is the two digit zero padded epoch number.

  • Replace the current version of VAST_PILOT.stmoc.fits in vasttools/data/mocs/ with the updated version containing the observations from the new epoch.

  • Add the new epoch to the RELEASED_EPOCHS variable found in vasttools.__init__.py.

  • If the new epoch is from RACS rather than VAST, add the new epoch to the RACS_EPOCHS variable in vasttools.__init__.py.

  • Make sure the new epoch data is present in the standard release format if the instance of VAST Tools has access to the survey data.

Warning: non-VAST data

Metadata from other surveys (e.g. RACS) is supported, and can be given arbitrary names. The only requirement is that the csv files must end in _info.csv and pickle files in _fields_csv.pickle. When adding data in this form the file prefix must also be added to the special_epoch_prefixes dict inside vasttools.survey._get_resource_path.

Example Epoch Addition

The following steps will allow users to add epoch 99 to the package, from start to finish assuming absolutely no pre-processing. In the future these steps may be simplified, for example if observations datetime data is already added to the fits images.

  1. Download the ASKAP_SURVEYS repo to /path/to/surveys/db/ and specify the base data directory with environment variable VAST_DATA_DIR. This assumes that the images of interest have already had the observation date added to them. If not, that can be done using vasttools.tools.add_obs_date.

  2. Run the following code to generate the fields info csv files.

Example: Generate epoch metadata

Generate all epoch metadata for epoch 99 

from vasttools.tools import create_fields_metadata
create_fields_metadata('99', '/path/to/surveys_db/')

  1. Move the resulting info csv file (vast_epoch99_info.csv) into vasttools/data/csvs/, and the resulting SkyCoord pickle file (vast_epoch99_fields_sc.pickle) into vasttools/data/pickles/

  2. Update the RELEASED_EPOCHS variable in vasttools.__init__.py, appending "99": "99" to the existing dictionary.

  3. If the fits files don't contain datetime information, add it with the following code. This should be done for all image types and polarisations, but only one is necessary for the following steps.

Example: Add datetime information to fits images

Add observation datetime information to stokes I combined images from epoch 99 

from vasttools.tools import add_obs_date
add_obs_date('99', 'COMBINED', 'STOKESI_IMAGES')

  1. Run the following code to generate all MOCs and STMOCs.

Example: Generate MOCs and STMOCs

Generate all MOCs and STMOCs for epoch 99 

from vasttools.tools import gen_mocs_epoch
gen_mocs_epoch('99', 'COMBINED', 'STOKESI_IMAGES')

  1. Move the full-epoch MOC, VAST_PILOT_EPOCH99.moc.fits, to vasttools/data/mocs/.

  2. Move the updated full-survey STMOC, VAST_PILOT.stmoc.fits, to vasttools/data/mocs/.

  3. Create a new git branch for this epoch with git checkout -b add-epoch-99.

  4. Commit all of the new changes, and push to the github repository with

Example: Push new epoch to git

Commit changes required for the addition of epoch 99 git add vasttools/data/csvs/vast_epoch99_info.csv git add vasttools/data/pickles/vast_epoch99_fields_sc.pickle git add __init__.py git add vasttools/data/mocs/VAST_PILOT_EPOCH99.moc.fits git add vasttools/data/mocs/VAST_PILOT.stmoc.fits

Last update: September 20, 2023
Created: October 21, 2021