Skip to content

Various fixes - #73

Merged
tofu-rocketry merged 12 commits into
apel:devfrom
tofu-rocketry:various-fixes
Dec 15, 2025
Merged

tofu-rocketry merged 12 commits into
apel:devfrom
tofu-rocketry:various-fixes

Conversation

@tofu-rocketry

@tofu-rocketry tofu-rocketry commented Dec 12, 2025 •

Copy link
Copy Markdown
Member

Resolves GT-1240
Resolves #63 (by speeding up the queries to an acceptable level)

This PR does the following:

  • Replace some of the queries with much faster ones.
  • Fixes some of the logic around which result is fetched for the benchmarks page so that you actually get the most relevant row for each site and submithost.
  • Removes redundant print() that were used when debugging during initial creation.
  • Fixes the comparisons for the status evaluation to allow future dates and add a critical level.
  • Tweaks the readme.
  • Also applies a small SQL change to the IRIS update script to try and improve its performance a bit.

Speed comparison

Before

Over 1 hour.

2025-11-27 07:01:02,050 - INFO - =====================
2025-11-27 07:01:11,655 - INFO - Refreshed GridSite
2025-11-27 07:01:31,630 - INFO - Refreshed GridSiteSync
2025-11-27 07:03:01,444 - INFO - Refreshed GridSiteSyncSubmitH
2025-11-27 07:17:13,359 - INFO - Refreshed CloudSite
2025-11-27 07:17:41,885 - INFO - Refreshed BenchmarksBySubmitHost from VSummaries
2025-11-27 08:17:42,443 - INFO - Refreshed BenchmarksBySubmitHost from VJobRecords
2025-11-27 08:17:51,911 - INFO - Refreshed BenchmarksBySubmitHost from VNormalisedSummaries
2025-11-27 08:17:51,911 - INFO - Data retrieval and processing attempted. Check the above logs for details on the sync status
2025-11-27 08:17:51,911 - INFO - =====================

After

Less than 5 minutes.

2025-12-12 20:47:16,860 - INFO - =====================
2025-12-12 20:47:19,483 - INFO - Refreshed GridSite
2025-12-12 20:47:32,894 - INFO - Refreshed GridSiteSync
2025-12-12 20:48:29,054 - INFO - Refreshed GridSiteSyncSubmitH
2025-12-12 20:49:06,731 - INFO - Refreshed CloudSite
2025-12-12 20:49:07,980 - INFO - Refreshed BenchmarksBySubmitHost from VSummaries
2025-12-12 20:51:31,971 - INFO - Refreshed BenchmarksBySubmitHost from VJobRecords
2025-12-12 20:51:32,210 - INFO - Refreshed BenchmarksBySubmitHost from VNormalisedSummaries
2025-12-12 20:51:32,210 - INFO - Data retrieval and processing attempted. Check the above logs for details on the sync status
2025-12-12 20:51:32,210 - INFO - =====================

- This version uses windowing (SELECT...ROW_NUMBER() OVER) and a smaller
  range of UpdateTime for efficiency.
- It uses a Common Table Expression (WITH...AS) for  readability.
- The first row for each site is then selected to give the latest values
  for that site.
- Counting the VMs is deprecated as it's not efficient to calculate and
  doesn't add much for the user. Removing it here should default the
  value to 0 in the model.
Previous commit removes the update of this data. We're deprecating it as
it doesn't add anything useful for the user at the moment and was
affecting the SQL performance.
- This version uses windowing (SELECT...ROW_NUMBER() OVER) and a smaller
  range of UpdateTime for efficiency.
- It uses a Common Table Expression (WITH...AS) for readability.
- The first row for each site and submit host is then selected to give
  the latest values for that site/submithost combination.
These changes are similar to teh Job Records imporvements, but
customised for the way these views work.
These were used during initial creation of the system to debug update
times, but aren't needed in production, particularly as we use a
completely different method to update the local Django database.
- Allow for future dates to be considered OK.
- Add a critical threshold.
@tofu-rocketry tofu-rocketry added this to the v1.5 milestone Dec 12, 2025
@tofu-rocketry tofu-rocketry self-assigned this Dec 12, 2025
@tofu-rocketry tofu-rocketry added the bug Something isn't working label Dec 12, 2025
@tofu-rocketry

tofu-rocketry commented Dec 12, 2025 •

Copy link
Copy Markdown
Member Author

TODO: Should improve the IRIS VAnonCloudRecords query - Done

@tofu-rocketry
tofu-rocketry marked this pull request as ready for review December 15, 2025 14:44

@rowan04 rowan04 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good, some very small comments

Comment thread monitoring/db_update_sqlite.py Resolved
Comment thread monitoring/db_update_sqlite.py Resolved
Comment thread monitoring/db_update_sqlite.py Resolved
Comment thread monitoring/db_update_sqlite.py Resolved
- Change last remaining CURRENT_TIMESTAMP to NOW().
- Replace DATE_SUB with plain subtraction of timestamp.

@rowan04 rowan04 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm :)

@tofu-rocketry
tofu-rocketry merged commit 4f17c53 into apel:dev Dec 15, 2025
1 check passed
@tofu-rocketry
tofu-rocketry deleted the various-fixes branch December 15, 2025 16:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Development

Successfully merging this pull request may close these issues.

Add "quick" option to DB update script

3 participants