Pipeline¶

This class provides methods to load and interact with results from the VAST Pipeline. There are ready made functions to:

• Load the pipeline outputs into dataframes ready for analysis.
• Explore individual sources.
• Run provided transient and variability searches.
• Check for the Sun, Moon and planets in the pipeline run.
• Recalculate the sources data after filters have been applied.
• Create a MOC of the pipeline run.

Using the Pipeline Component¶

The first step is to initialise a Pipeline instance from vasttools.pipeline:

from vasttools.pipeline import Pipeline

pipe = Pipeline()

If configured correctly, VAST Tools should automatically detected where the Pipeline directory is located on the system. If this is not the case then the variable project_dir can be passed when initialising the instance to define where the pipeline outputs are located. Refer to the Configuration & Data Access page for more information.

pipe = Pipeline(project_dir='/path/to/the/pipeline-runs/')

Available Pipeline Methods¶

The following methods are available with the Pipeline instance.

list_images¶

Code reference.

Provides a list of all the images that have been processed in the pipeline.

pipe.list_images()

 ['VAST_2118+00A_EPOCH01_I_fits',
 'VAST_2118-06A_EPOCH01_I_fits',
 'VAST_2118-06A_EPOCH02_I_fits',
 'VAST_2118-06A_EPOCH03x_I_fits',
 'VAST_2118-06A_EPOCH05x_I_fits',
 'VAST_2118-06A_EPOCH06x_I_fits']

list_piperuns¶

Code reference.

Provides a list of all the pipeline runs that have been set up in the pipeline.

pipe.list_piperuns()

['S190814bv',
 'VAST_0127-73A',
 'VAST_2118-06A_UI']

load_run¶

Code reference.

Loads the pipeline run requested returning a vasttools.pipeline.PipeAnalysis instance (see sections below).

my_pipe_run = pipe.load_run('pilot_survey')

load_runs¶

Code reference.

Loads multiple pipeline runs that are requested and merges them to return a single vasttools.pipeline.PipeAnalysis instance (see sections below).

my_pipe_run = pipe.load_runs(['pilot_survey', 'pilot_survey_phase_2'])

PipeAnalysis Instances¶

A loaded run using the Pipeline.load_run() method is returned as a vasttools.pipeline.PipeAnalysis instance. This object contains the loaded raw data of the pipeline as well has having access to various methods to perform common analysis tasks. The sections below detail what data and methods are available.

Accessing PipeAnalysis Run Data¶

Code reference.

The run data is stored as attributes to the PipeAnalysis object, which are all shown in the table below. This means that to access any of the pieces of pipeline run data, for example sources, the command would be:

my_pipe_run.sources

Available PipeAnalysis Methods¶

The following methods are available to the PipeAnalysis instance.

check_for_planets¶

Code reference.

Checks to see if any of the planets, including Pluto, the Sun and the Moon, are observable at any time in the pipeline run. A pandas dataframe will be returned containing the positions of found objects. An empty dataframe will be returned if no objects are in the pipeline run.

The columns of the returned dataframe are:

• DATEOBS The date of the observation (image) being checked.
• centre-ra The right ascension coordinate of the image centre in degrees.
• centre-dec The declination coordinate of the image centre in degrees.
• planet The name of the planet.
• ra The right ascension coordinate of the planet location in degrees.
• dec The declination coordinate of the planet location in degrees.
• sep The on-sky separation between the planet location and the image centre in degrees.

planets = my_pipe_run.check_for_planets()

combine_with_run¶

Code reference.

Combines the current run with another PipeAnalysis object that has been loaded, i.e. a second pipeline run. It returns a new PipeAnalysis object with the two runs combined.

my_new_pipe_run = my_pipe_run.combine_with_run(my_second_pipe_run)

create_moc¶

Code reference.

This creates a Multi-Order Coverage map for the area covered by the pipeline run.

my_pipe_run_moc = my_pipe_run.create_moc()

filter_by_moc¶

Code reference.

Returns a new PipeAnalysis instance that has had all data filtered to only include data of sources that are within a supplied MOC.

my_pipe_run_filtered = my_pipe_run.filter_by_moc(my_moc)

get_source¶

Code reference.

Fetches a specific source from the pipeline run, and returns a vasttools.source.Source instance. The source is selected by using the ID value.

my_source = my_pipe_run.get_source(500)

get_sources_skycoord¶

Code reference.

A convenience function to generate a SkyCoord object of all the sources in the PipeAnalysis.sources attribute. Returns a astropy.coordines.SkyCoord instance.

sources_skycoord = my_pipe_run.get_sources_skycoord()

load_two_epoch_metrics¶

Code reference.

By default the raw two-epoch metrics produced by the pipeline are not loaded. This method loads the two-epoch metrics, stored by the pipeline in measurement_pairs.parquet (or measurement_pairs.arrow) and saves them as an attribute to the PipeAnalysis object: PipeAnalysis.measurement_pairs_df. The method also creates a second dataframe that lists all the possible epoch pairs, i.e., all the unique pairs of images that make up the measurement pairs. This is stored as the attribute PipeAnalysis.pairs_df.

my_pipe_run.load_two_epoch_metrics()

my_pipe_run.measurement_pairs_df
my_pipe_run.pairs_df

recalc_measurement_pairs_df¶

Code reference.

A method to recalculate the measurement pairs dataframe using a provided measurements dataframe that is assumed to have been edited. For example, some measurements could have been removed or the fluxes could have been changed to corrected values.

new_measurement_pairs = my_pipe_run.recalc_measurement_pairs_df(my_filtered_measurements)

recalc_sources_df¶

Code reference.

This method recalculates the sources dataframe using the provided measurements dataframe. In particular, all the columns that are averages, counts or the variability metrics are recalculated. It is useful for cases where measurements are filtered out of the original measurements dataframe, or have fluxes changed, and hence the source dataframe becomes out of sync. Returns a pandas dataframe.

new_sources_df = my_pipe_run.recalc_sources_df(my_filtered_measurements)

new_measurement_pairs = my_pipe_run.recalc_measurement_pairs_df(my_filtered_measurements)

new_sources_df = my_pipe_run.recalc_sources_df(
    my_filtered_measurements, measurement_pairs_df=new_measurement_pairs)

my_pipe_run.sources = new_sources_df

Available PipeAnalysis Transient Analysis Methods¶

The following methods are designed to perform common transient search analyses and plots on the loaded pipeline run data, including the production of useful plots.

eta_v_diagnostic_plot¶

Code reference.

Produces a diagnostic plot using the η and V metrics as described in Rowlinson et al., 2019. Returns a matplotlib.pyplot.figure instance.

diag_plot = my_pipe_run.eta_v_diagnostic_plot()

Example Plot

plot_two_epoch_pairs¶

Code reference.

Produce a two-epoch metric plot of the m and vs values of the requested pair ID (i.e. the pair of images). The pair ID can be found in the pairs_df attribute after load_two_epoch_metrics() has been used. There are options to change the m_min and vs_min thresholds and to switch to using integrated flux instead of the peak flux (default). The plots can be returned as a matplotlib.pyplot.figure instance or a bokeh.plotting.figure instance. Also available are two styles, style a follows the plot style found in Mooley et al., 2016 where as style b follows the style found in Radcliffe et al., 2019.

from bokeh.io import output_notebook

output_notebook()

two_epoch_plot = my_pipe_run.plot_two_epoch_pairs(14)

Example Plot Style a

Example Plot Style b

run_eta_v_analysis¶

Code reference.

Method to perform the η-V transient search method as described in Rowlinson et al., 2019, to the sigma level provided for each metric. It uses the η and V metrics defined in the sources dataframe. The distributions of η and V in the sources dataframe are fitted with a Gaussian, which is then used to calculate the requested sigma value for each parameter. Any sources that have η and V values above the calculated thresholds are considered as variable candidates.

The method returns a tuple containing:

• η cutoff value,
• the V cutoff value,
• dataframe of candidates,
• candidates plot,
• the diagnostic plot (optional).

The η and V cutoff values are the calculated values for the provided sigma limit. The candidates dataframe contains any sources that exceed both the η and V thresholds (found in the top right-hand quadrant of the plot). The candidates plot can either be a matplotlib.pyplot.figure or bokeh.layouts.gridplot instance.

my_query_string = (
    "n_measurements >= 3 "
    "& n_relations == 0 "
    "& min_snr >= 7.0"
)

eta_thresh, v_thresh, eta_v_candidates, plot = my_run.run_eta_v_analysis(1.0, 1.0, query=my_query_string)

eta_thresh, v_thresh, eta_v_candidates, plot = my_run.run_eta_v_analysis(1.0, 1.0)

Example Plot

run_two_epoch_analysis¶

Code reference.

Method to perform the two epoch analysis as detailed in Mooley et al., 2016 to the provided m and vs threshold values. The entire measurement_pairs_df is analysed for any pair that exceed the two provided thresholds.

The method returns a tuple containing two dataframes: the candidate sources and pairs. The candidate sources are the unique sources that have at least one measurement pair that exceed the threshold. The candidate pairs are all the measurement pairs that exceed the provided thresholds.

two_epoch_candidate_sources, two_epoch_candidate_pairs = my_run.run_two_epoch_analysis(4.3, 0.26)