Benchmarks¶

Initial Profiling¶

These profiling tests were run with the pipeline codebase correspondent to commit 373c2ce (some commits after the first release).

Running on 12GB of data with 464MB peak memory usage takes 4 mins: performance: ~80% final_operations, ~10% get_src_skyregion_merged_df. final_operations calls other functions, out of these the largest is 50% in make_upload_sources which spends about 20% of time on utils <method 'execute' of 'psycopg2.extensions.cursor' objects>. The get_src_skyregion_merged_df time sink is in threading 15% of time is on wait (final_operations spends some time on threading as well, hence 15% > 10%). memory: 40% pyarrow parquet write_table, rest is mostly fragmented, some more pyarrow and some pandas

Running on 3MB of data with peak memory usage 176MB, takes 1.5s: performance: ~30% goes to pipeline (about 9% of this is pickle), ~11% goes to read, rest goes to django I think memory: 30% memory is spent on django, 20% is spent on astropy/coordinates/matrix_utilities, 10% on importing other modules, rest is fragmented quite small

Note that I didn't include the generation of the images/*/measurements.parquet or other files in these profiles.

Database Update Operations¶

Delete (Model.objects.all().delete()) and reupload (bulk_upload) (in seconds)

Per cell, 10³ rows is slower than 10⁴ and 10⁵ rows, possibly due to overhead. Best to avoid uploading 10³ rows each bulk_create call.

Django bulk_update

I don't think there's any point testing 10⁴ or 10⁵ rows, it's obviously the worst performing function, and I've already had to force quit the terminal twice because keyboard interrupt didn't work.

SQL join as (SQL_update in vast_pipeline.pipeline.loading)

10⁵ is slower per cell than 10⁴ and 10³, not sure why. Recommend updating 10⁴ rows each time.

This timing info does vary a bit on randomness. Sometimes the SQL join as takes as long as 1 second to complete 10³ rows, I'm not sure what's causing this.