Skip to content

Pipeline Local Development Environment Guidelines

This section describes how to set up a local development environment more in details.

Back End

Installation

The installation instructions are the same as the ones describes in the Getting Started section with one key difference. Rather than installing the Python dependencies with pip, you will need to install and use Poetry. After installing Poetry, running the command below will install the pipeline dependencies defined in poetry.lock into a virtual environment. The main difference between using Poetry and pip is that pip will only install the dependencies necessary for using the pipeline, whereas Poetry will also install development dependencies required for contributing (e.g. tools to build the documentation).

poetry install

Note

Poetry will automatically create a virtual environment if it detects that your shell isn't currently using one. This should be fine for most users. If you prefer to use an alternative virtual environment manager (e.g. Miniconda), you can prevent Poetry from creating virtual environments. However, even if you are using something like Miniconda, allowing Poetry to manage the virtualenv (the default behaviour) is fine. The development team only uses this option during our automated testing since our test runner machines only contain a single Python environment so using virtualenvs is redundant.

Changes to the data models

When changes are made to the data models defined in vast_pipeline/models.py, the database schema needs to be updated to reflect those changes. Django can handle this for you by generating migration files that contain the necessary code to update the schema. Migration files are generated with the Django management command python manage.py makemigrations and applied to the database with python manage.py migrate. Depending on the nature of the changes, this may break backward compatibility, i.e. runs created with previous versions of the pipeline may not be compatible with your changes.

Database migrations must be committed to source control so that others can pull in your model changes. They can sometimes be complex and require additional attention. If you have any difficulty with migrations, please contact the VAST development team for help. More information can be found in the Django documentation on migrations.

Removing/Clearing Data

The following sub-sections show how to completely drop every data in the database and how to remove only the data related to one or more pipeline runs.

Reset the database

Make sure you installed the requirements dev.txt. And django_extensions is in EXTRA_APPS in your setting configuration file .env (e.g. EXTRA_APPS=django_extensions,another_app,...).

(pipeline_env)$ ./manage.py reset_db  && ./manage.py migrate
# use the following for no confirmation prompt
(pipeline_env)$ ./manage.py reset_db --noinput  && ./manage.py migrate

Clearing Run Data

It is sometimes convenient to remove the data belonging to one or more pipeline runs while developing the code base. This is particularly useful to save time by not having to re-upload the image data along with the measurements. The data related to the pipeline are the Sources, Associations, Forced extractions entries in database and the parquet files in the respective folder. By default the command will keep the run folder with the config and the log files.

(pipeline_env)$ ./manage.py clearpiperun path/to/my-pipe-run

To clear more than one run:

(pipeline_env)$ ./manage.py clearpiperun path/to/my-pipe-run1 my-pipe-run2 path/to/my-pipe-run3

The command accept both a path or a name of the pipeline run(s). To remove all the runs, issue:

(pipeline_env)$ ./manage.py clearpiperun clearall

The command to keep the parquet files is:

(pipeline_env)$ ./manage.py clearpiperun path/to/my-pipe-run --keep-parquet

The remove completely the pipeline folder

(pipeline_env)$ ./manage.py clearpiperun path/to/my-pipe-run --remove-all

FrontEnd Assets Management and Guidelines

This guide explain the installation, compilation and development of the front end assets (HTML, CSS, JS and relative modules). We make use of a node installation with npm and gulp tasks to build the front end assets.

Installation of node packages

After installing a node version and npm, install the node modules using from the base folder (vast-pipeline):

npm ci

Npm will install the node packages in a node_modules folder under the main root.

...
├── node_modules
...

For installing future additional dependencies you can run npm install --save my-package or npm install --save-dev my-dev-package (to save a development module), and after that commit both package.json and package-lock.json files. For details about the installed packages and npm scripts see package.json.

FrontEnd Tasks with gulp

Using gulp and npm scripts you can:

  1. Install dependencies under the ./static/vendor folder.
  2. Building (e.g. minify/uglify) CSS and/or Javascript files.
  3. Run a development server that "hot-reload" your web page when any HTML, CSS or Javascript file is modified.

The command to list all the gulp "tasks" and sub-tasks is (you might need gulp-cli installed globally, i.e. npm i --global gulp-cli, more info here):

gulp --tasks

Output:

[11:55:30] Tasks for ~/PATH/TO/REPO/vast-pipeline/gulpfile.js
[11:55:30] ├── clean
[11:55:30] ├─┬ js9
[11:55:30] │ └─┬ <series>
[11:55:30] │   ├── js9Dir
[11:55:30] │   ├── js9MakeConfig
[11:55:30] │   ├── js9Make
[11:55:30] │   ├── js9MakeInst
[11:55:30] │   └── js9Config
[11:55:30] ├── css
[11:55:30] ├── js
[11:55:30] ├─┬ vendor
[11:55:30] │ └─┬ <parallel>
[11:55:30] │   ├── modules
[11:55:30] │   └─┬ <series>
[11:55:30] │     ├── js9Dir
[11:55:30] │     ├── js9MakeConfig
[11:55:30] │     ├── js9Make
[11:55:30] │     ├── js9MakeInst
[11:55:30] │     └── js9Config
[11:55:30] ├─┬ build
[11:55:30] │ └─┬ <series>
[11:55:30] │   ├─┬ <parallel>
[11:55:30] │   │ ├── modules
[11:55:30] │   │ └─┬ <series>
[11:55:30] │   │   ├── js9Dir
[11:55:30] │   │   ├── js9MakeConfig
[11:55:30] │   │   ├── js9Make
[11:55:30] │   │   ├── js9MakeInst
[11:55:30] │   │   └── js9Config
[11:55:30] │   └─┬ <parallel>
[11:55:30] │     ├── cssTask
[11:55:30] │     └── jsTask
[11:55:30] ├─┬ watch
[11:55:30] │ └─┬ <series>
[11:55:30] │   ├─┬ <parallel>
[11:55:30] │   │ ├── cssTask
[11:55:30] │   │ └── jsTask
[11:55:30] │   └─┬ <parallel>
[11:55:30] │     ├── watchFiles
[11:55:30] │     └── browserSync
[11:55:30] ├─┬ default
[11:55:30] │ └─┬ <series>
[11:55:30] │   ├─┬ <parallel>
[11:55:30] │   │ ├── modules
[11:55:30] │   │ └─┬ <series>
[11:55:30] │   │   ├── js9Dir
[11:55:30] │   │   ├── js9MakeConfig
[11:55:30] │   │   ├── js9Make
[11:55:30] │   │   ├── js9MakeInst
[11:55:30] │   │   └── js9Config
[11:55:30] │   └─┬ <parallel>
[11:55:30] │     ├── cssTask
[11:55:30] │     └── jsTask
[11:55:30] └── debug

Alternatively you can run gulp from the installed version in the node_modules folder with:

./node_modules/.bin/gulp --tasks

For further details about tasks, see gulpfile.

1. Install Dependencies under vendor Folder

Install the dependencies under the ./static/vendor folder, with:

npm run vendor

Or, using global gulp-cli:

gulp vendor

As seen in the tasks diagram above, the vendor task run the module task in parallel with the js9 tasks. JS9 has many task as these run with manual command that involve make/make install and then writing configuration to js9prefs.js file. You can run manually the installation of JS9 with gulp js9.

2. Building CSS and Javascript files

npm run build
# or
npm start
# or
gulp build
# or
gulp default
# or
gulp

will run the vendor task and minify both CSS and Javascript files. By default, when no other tasks is specified, gulp runs the build task. You can run single tasks with:

gulp css

to run just the minification of the CSS files.

3. Run Development Server

Start your normal Django server with (NOTE: do not change the default port!):

(pipeline_env)$: ./manage.py runserver

In another terminal run:

npm run watch
# or
gulp watch

The latter will open your dev server, that will auto reload and apply your latest changes in any CSS, Javascript and/or HTML files. As pointed out in the gulp task tree above the watch task run both the vendor and build tasks.

4. Debug Task

This task is for debugging the paths used in the others task, but also serve as a place holder to debug commands.

npm run debug
# or
gulp debug

5. Clean Task

This task delete the vendor folder (/static/vendor) along with all the files.

npm run clean
# or
gulp clean

FrontEnd assets for Production

In order to compile the frontend assets for production, activate the Python virtual environment, then run:

(pipeline_env)$ npm run js9staticprod && ./manage.py collectstatic -c

This command will collect all static assets (Javascript and CSS files) and copy them to STATIC_ROOT path in setting.py, so make sure you have permission to write to that. STATIC_ROOT is assigned to ./staticfiles by default, otherwise assigned to the path you defined in your .env file.

The js9staticprod gulp task is necessary if you specify a STATIC_URL and a BASE_URL different than the default, for example if you need to prefix the site / with a base url because you are running another webserver (e.g. another web server is running on https://my-astro-platform.com/ so you want to run the pipeline on the same server/domain https://my-astro-platform.com/pipeline, so you need to set BASE_URL='/pipeline/' and STATIC_URL=/pipeline-static/ in settings.py). We recommend to run this in any case!

Then you can move that folder to where it can be served by the production static files server (Ningx or Apache are usually good choices, in case refer to the Django documentation).