For project documentation, please refer to https://github.com/umccr/orcahouse-doc
For data warehouse issue tracking, please submit an issue to https://github.com/umccr/orcahouse-project/issues
Create a Python virtual environment (any method) and install the dev toolchain requirements.
See README_DEV.md for more comprehensive setup details.
conda activate orcavault
make install
make check
We need an authenticated AWS session as we are developing against the remote OrcaVault dev environment.
Use your usual AWS CLI setup to authenticate.
For example:
export AWS_PROFILE=unimelb-warehouse-prod-poweruser
aws sso login
Source the dx script. It exports shell functions (command shortcut) related to the remote OrcaVault dev environment.
source dx.sh
Connect to the dev warehouse. This makes the house key, the hosts entry, the tunnel and the dbt credentials.
house_up
Each step is also its own function: house_key, house_host, house_tunnel, house_nc and house_cred. house_down
closes the tunnel and clears the credentials.
Run the dbt debug command to check the connection.
dbt debug
You should expect to see a successful connection.
<...>
07:22:44 Registered adapter: redshift=1.10.1
07:22:47 Connection test: [OK connection ok]
07:22:47 All checks passed!
Building in your own personal schema is the preferred workflow. It's named after your AWS SSO username by default, run
house_schema to see yours, or set ORCAVAULT_SCHEMA to pick another name. It works like a diff over the dev daily
run, which you sync with using dbt_defer or dbt_clone before building. See
README_DEV.md for setup and instructions.
If you want to opt out of personal schemas, run house_shared after house_up to go back to the original workflow
using the shared dev schema, in which case the dbt section below applies.
The dbt has multiple targets. You can dbt --help for more details.
dbt debug
dbt clean
dbt deps
dbt compile
dbt build
dbt seed
dbt run
dbt test
You can run a specific model by name.
dbt run -s hub_sequencing_run
Running the incremental model won't be updated upon the consecutive run. You can pass --full-refresh flag to reload
from the beginning all over again. BUT. Doing so will lose the model's historical load_datetime history.
dbt run -s hub_sequencing_run --full-refresh
The append-only ICA usage models ignore a project-wide --full-refresh, so a routine full refresh of the project cannot
silently discard their load history. A column added to a model's SQL is applied to the existing table in place with
on_schema_change: append_new_columns, so widening these models never needs a rebuild. Rebuild them only when you mean
it, naming them explicitly:
dbt run -s spreadsheet__ica_usage_report sat_workflow_run_ica_usage --vars '{ica_usage_full_refresh: true}'
This is lossless only while tsa.spreadsheet__ica_usage_report still carries every source row.
- Login to the Data Warehouse AWS account console using
AWSPowerUserAccessrole. - Navigate to the Redshift QueryEditor v2.
- Select the
Serverless: orcahouse-devwarehouse for development. - On the connection prompt,
- Select
Federated User - Enter
orcavaultat database name - Click
Create Connectionbutton
- Select