Installation¶

This document provides instructions on installing the VAST Pipeline for local use.

The VAST Pipeline consists of 3 main components that require installation:

a PostgreSQL database,
a Django application,
a front-end website.

The instructions have been tested on Debian/Ubuntu and macOS.

PostgreSQL¶

We recommend using a Docker container for the database rather than installing the database system-wide.

Steps:

Install Docker. Refer to the official documentation, and for Ubuntu users to this. Remember to add your user account to the docker group official docs, by running:
```
sudo groupadd docker
sudo usermod -aG docker $USER
```
Create a PostgreSQL container. The VAST Pipeline requires a PostgreSQL database with the Q3C plugin to enable special indexing on coordinates and fast cone-search queries. We have prepared a Docker image based on the latest PostgreSQL image that includes Q3C . Start a container using this image by running the command below, replacing <container-name> with a name of your choice (e.g. vast-pipeline-db) and <password> with a password of your choice which will be set for the default postgres database superuser account.
```
docker run --name <container-name> --env POSTGRES_PASSWORD=<password> --publish-all --detach ghcr.io/marxide/postgres-q3c:latest
```
The --publish-all option will make the PostgreSQL server port 5432 in the container accessible on a random available port on your system (the host). The --detach option instructs Docker to start the container in the background rather than taking over your current shell. Verify that the container is running and note the host port that 5432/tcp is published on by running docker ps, e.g. in the example below, the host port is 55002.
```
docker ps
CONTAINER ID   IMAGE                                 COMMAND                  CREATED         STATUS         PORTS                     NAMES
8ff553add2ed   ghcr.io/marxide/postgres-q3c:latest   "docker-entrypoint.s…"   4 seconds ago   Up 3 seconds   0.0.0.0:55002->5432/tcp   vast-pipeline-db
```

The database server should now be running in a container on your machine.

Tip

To stop the database server, simply stop the container with the following command

docker stop <container-name or container-id>

You can start an existing stopped container with the following command

docker start <container-name or container-id>

Note that docker run and docker start are not the same. docker run will create and start a container from an image; docker start will start an existing stopped container. If you have previously created a VAST Pipeline database container and you wish to reuse it, you want to use docker start. You will likely need to restart the container after a system reboot.

Python Environment¶

We strongly recommend installing the VAST Pipeline in an isolated virtual environment (e.g. using Miniconda, Virtualenv, or venv). This will keep the rather complex set of dependencies separated from the system-wide Python installation.

Create a new Python environment using your chosen virtual environment manager and activate it. For example, Miniconda users should run the following command, replacing <environment-name> with an appropriate name (e.g. pipeline-env):
```
conda create --name <environment-name> python=3.8
conda activate <environment-name>
```
Note

All further installation instructions will assume you have activated your new virtual environment. Your environment manager will usually prepend the virtual environment name to the shell prompt, e.g.
```
(pipeline-env)$ ...
```
Clone the pipeline repository https://github.com/askap-vast/vast-pipeline and change into the repo directory.
```
git clone https://github.com/askap-vast/vast-pipeline.git
cd vast-pipeline
```
Warning

Do not change the the repo folder name, e.g. git clone https://github.com/askap-vast/vast-pipeline.git my-pipeline-local-dev
(Optional) Checkout the version you want to install. Currently, the repo will have cloned the latest code from the master branch. If you require a specific version, checkout the appropriate version tag into a new branch e.g. for version 0.2.0
```
git checkout -b <new-branch-name> 0.2.0
```
Install non-Python dependencies. Some of the Python dependencies required by the pipeline depend on some non-Python libraries. These can also be installed by Miniconda, otherwise they are best installed using an appropriate package manager for your operating system e.g. apt for Debian/Ubuntu, dnf for RHEL 8/CentOS 8, Homebrew for macOS. The dependencies are:
MinicondaDebian/UbuntuRHEL/CentOSHomebrew
- libpq
- graphviz
Both are available on the conda-forge channel. They are also specified in the environment file requirements/environment.yml which can be used to install the required packages into an activated conda environment with the following command
conda env update -f requirements/environment.yml
- libpq-dev
- libgraphviz-dev
- libpq-devel
- graphviz-devel
CentOS users

You may need to enable the PowerTools repository to install graphviz-devel.

dnf install dnf-plugins-core dnf config-manager --set-enabled powertools
- libpq
- graphviz
Install the pipeline and it's Python dependencies.
```
pip install .
```
Warning

Don't forget the . at the end of the above command. It instructs pip that the root directory of the package to install is the current directory.

Tip

If you are intending to deploy an instance of the pipeline onto a server, you may also want to install the recommended production extras with pip install .[prod]. However, note that these are recommendations only and there are other alternative packages that may work just as well.

Tip

If you intend to contribute to development of the pipeline, you will need the Python dependency management tool Poetry. See the development guidelines.

Front-End Assets Quickstart¶

In order to install and compile the front-end website assets (modules like js9 and bootstrap, as well as minification of JS and CSS files) you need a recent version of NodeJS installed.

Installation of NodeJS¶

If you are using Miniconda and installed the requirements/environment.yml file as shown above, then NodeJS is already installed. Otherwise, we recommend following the instructions on the NodeJS downloads page for your OS (there are many installation options).

Setting up the front-end assets¶

In order to set up the front end assets, run:

npm ci && npm start

Note

Ensure you are still in the root of the repo before running the command above. The npm ci command ("clean install") will remove all previous node modules and install all the dependencies from scratch. The npm start command will run the default gulp "task" which, among other things, compiles Sass into CSS, minifies CSS and JS files, and copies these files into the static/vendor folder. For more details of compilation of frontend assets (e.g. single tasks), and front-end developement set up read the Front End Developing Guidelines.

Bug

When npm start or npm run start was run in a Ubuntu 20.04 LTS (containerised environment), for some unknown reasons, both commands failed with the following error.

[12:48:19] 'js9Make' errored after 7.67 ms
[12:48:19] Error: spawn make ENOENT
    at Process.ChildProcess._handle.onexit (internal/child_process.js:267:19)
    at onErrorNT (internal/child_process.js:469:16)
    at processTicksAndRejections (internal/process/task_queues.js:84:21)
[12:48:19] 'default' errored after 2.63 s
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! vast-pipeline@99.99.99-dev start: `gulp default`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the vast-pipeline@99.99.99-dev start script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
npm ERR! A complete log of this run can be found in:
npm ERR!     /home/vast/.npm/_logs/2020-10-06T01_48_19_215Z-debug.log

The way around for this issue is unorthodox. The following steps were followed to overcome the issue:

cd node_modules/js9/
./configure
make
make install
cd ~/vast-pipeline/  ## (to comeback to the root folder of the project)
npm install

That somehow solved the issue mentioned above.

Done! Now go to Vast Pipeline Configuration file to see how to initialize and run the pipeline. Otherwise if you intend on developing the repo open the Contributing and Developing Guidelines file for instructions on how to contribute to the repo.