Database maintenance mode enabled

Databases undergo regular routine maintenance. Find out more about what database maintenance means.

Job request: 24242

Organisation:: University of Bristol
Workspace:: metformin-covid-main
ID:: zvabv67ktqyqzrk5

This page shows the technical details of what happened when the authorised researcher Alain Amstutz requested one or more actions to be run against real patient data within a secure environment.

By cross-referencing the list of jobs with the pipeline section below, you can infer what security level the outputs were written to.

The output security levels are:

highly_sensitive
- Researchers can never directly view these outputs
- Researchers can only request code is run against them
moderately_sensitive
- Can be viewed by an approved researcher by logging into a highly secure environment
- These are the only outputs that can be requested for public release via a controlled output review service.

Jobs

Action:

generate_dataset

Status:

Status: Succeeded

Job identifier:

jy6xukcsxabvedby
Action:

data_process

Status:

Status: Succeeded

Job identifier:

cdcy7uvlmtez3qhn
Action:

km_estimates_agi_mono

Status:

Status: Succeeded

Job identifier:

bdpu5opwos3oo27z
Action:

km_estimates_dpp4_mono

Status:

Status: Succeeded

Job identifier:

piodcuks6ejrwocz
Action:

km_estimates_glp1_mono

Status:

Status: Succeeded

Job identifier:

hzzozrxpdpoxhztn
Action:

km_estimates_insulin_mono

Status:

Status: Succeeded

Job identifier:

k36kdqvm3cbmqdaf
Action:

km_estimates_megli_mono

Status:

Status: Succeeded

Job identifier:

n2champg2f5wzisu
Action:

km_estimates_metfin

Status:

Status: Succeeded

Job identifier:

ebrai4ct2dpbgphr
Action:

km_estimates_metfin_mono

Status:

Status: Succeeded

Job identifier:

m32s4jil6womqfrr
Action:

km_estimates_sglt2_mono

Status:

Status: Succeeded

Job identifier:

k7ntoqwqjfwhnj4i
Action:

km_estimates_sulfo_mono

Status:

Status: Succeeded

Job identifier:

6ahfrgqed4ihp3vy
Action:

km_estimates_tzd_mono

Status:

Status: Succeeded

Job identifier:

g3hygjc4xuzpcotn
Action:

plot_km_estimates

Status:

Status: Succeeded

Job identifier:

4ttdbtta7tjcczo2

Pipeline

Show project.yaml

version: '3.0'

# Ignore this`expectation` block. It is required but not used, and will be removed in future versions.
expectations:
  population_size: 1000

actions:
  study_dates:
    run: r:latest analysis/metadates.R
    outputs:
      highly_sensitive:
        study_dates_json: output/study_dates.json

  generate_dataset_dm_algo:
    run: ehrql:v1 generate-dataset analysis/dataset_definition_dm_algo.py --output output/dataset_dm_algo.arrow
    needs: 
    - study_dates
    outputs:
      highly_sensitive:
        dataset: output/dataset_dm_algo.arrow
  
  data_process_dm_algo:
    run: r:latest analysis/data_process_dm_algo.R
    needs:
    - generate_dataset_dm_algo
    outputs:
      highly_sensitive:
        dataset: output/data/data_processed_dm_algo.csv.gz

  #generate_dataset:
  #  run: ehrql:v1 generate-dataset analysis/dataset_definition_landmark.py --output output/dataset.arrow
  #  needs: 
  #  - study_dates
  #  outputs:
  #    highly_sensitive:
  #      dataset: output/dataset.arrow

  generate_dataset:
    run: ehrql:v1 generate-dataset analysis/dataset_definition_t2dm.py --output output/dataset.arrow
    needs: 
    - study_dates
    - generate_dataset_dm_algo
    - data_process_dm_algo
    outputs:
      highly_sensitive:
        dataset: output/dataset.arrow
  
  #data_process:
  #  run: r:latest analysis/data_process.R
  #  needs:
  #  - generate_dataset
  #  - data_process_dm_algo
  #  outputs:
  #    highly_sensitive:
  #      rds: output/data/data_processed.rds
  #      feather: output/data/data_plots.feather
  #    moderately_sensitive:
  #      csv: output/data_properties/*.csv

  data_process:
    run: r:latest analysis/data_process.R
    needs:
    - generate_dataset
    outputs:
      highly_sensitive:
        rds: output/data/data_processed.rds
        feather: output/data/data_plots.feather
      moderately_sensitive:
        csv: output/data_properties/*.csv

  km_estimates_metfin:
    run: r:latest analysis/km.R 
      --df_input=output/data/data_plots.feather
      --dir_output=output/metfin
      --exposure=exp_bin_metfin_anytime
      --origin_date=elig_date_t2dm
      --event_date=exp_date_metfin_anytime
      --censor_date=out_date_severecovid
      --max_fup=730
      --plot=FALSE
    needs:
    - data_process
    outputs:
      moderately_sensitive:
        output: output/metfin/*.csv
  
  km_estimates_metfin_mono:
    run: r:latest analysis/km.R 
      --df_input=output/data/data_plots.feather
      --dir_output=output/metfin_mono
      --exposure=exp_bin_metfin_mono_anytime
      --origin_date=elig_date_t2dm
      --event_date=exp_date_metfin_mono_anytime
      --censor_date=out_date_severecovid
      --max_fup=730
      --plot=FALSE
    needs:
    - data_process
    outputs:
      moderately_sensitive:
        output: output/metfin_mono/*.csv

  km_estimates_dpp4_mono:
    run: r:latest analysis/km.R 
      --df_input=output/data/data_plots.feather
      --dir_output=output/dpp4_mono
      --exposure=exp_bin_dpp4_mono_anytime
      --origin_date=elig_date_t2dm
      --event_date=exp_date_dpp4_mono_anytime
      --censor_date=out_date_severecovid
      --max_fup=730
      --plot=FALSE
    needs:
    - data_process
    outputs:
      moderately_sensitive:
        output: output/dpp4_mono/*.csv

  km_estimates_tzd_mono:
    run: r:latest analysis/km.R 
      --df_input=output/data/data_plots.feather
      --dir_output=output/tzd_mono
      --exposure=exp_bin_tzd_mono_anytime
      --origin_date=elig_date_t2dm
      --event_date=exp_date_tzd_mono_anytime
      --censor_date=out_date_severecovid
      --max_fup=730
      --plot=FALSE
    needs:
    - data_process
    outputs:
      moderately_sensitive:
        output: output/tzd_mono/*.csv

  km_estimates_sglt2_mono:
    run: r:latest analysis/km.R 
      --df_input=output/data/data_plots.feather
      --dir_output=output/sglt2_mono
      --exposure=exp_bin_sglt2_mono_anytime
      --origin_date=elig_date_t2dm
      --event_date=exp_date_sglt2_mono_anytime
      --censor_date=out_date_severecovid
      --max_fup=730
      --plot=FALSE
    needs:
    - data_process
    outputs:
      moderately_sensitive:
        output: output/sglt2_mono/*.csv

  km_estimates_sulfo_mono:
    run: r:latest analysis/km.R 
      --df_input=output/data/data_plots.feather
      --dir_output=output/sulfo_mono
      --exposure=exp_bin_sulfo_mono_anytime
      --origin_date=elig_date_t2dm
      --event_date=exp_date_sulfo_mono_anytime
      --censor_date=out_date_severecovid
      --max_fup=730
      --plot=FALSE
    needs:
    - data_process
    outputs:
      moderately_sensitive:
        output: output/sulfo_mono/*.csv

  km_estimates_glp1_mono:
    run: r:latest analysis/km.R 
      --df_input=output/data/data_plots.feather
      --dir_output=output/glp1_mono
      --exposure=exp_bin_glp1_mono_anytime
      --origin_date=elig_date_t2dm
      --event_date=exp_date_glp1_mono_anytime
      --censor_date=out_date_severecovid
      --max_fup=730
      --plot=FALSE
    needs:
    - data_process
    outputs:
      moderately_sensitive:
        output: output/glp1_mono/*.csv

  km_estimates_megli_mono:
    run: r:latest analysis/km.R 
      --df_input=output/data/data_plots.feather
      --dir_output=output/megli_mono
      --exposure=exp_bin_megli_mono_anytime
      --origin_date=elig_date_t2dm
      --event_date=exp_date_megli_mono_anytime
      --censor_date=out_date_severecovid
      --max_fup=730
      --plot=FALSE
    needs:
    - data_process
    outputs:
      moderately_sensitive:
        output: output/megli_mono/*.csv

  km_estimates_agi_mono:
    run: r:latest analysis/km.R 
      --df_input=output/data/data_plots.feather
      --dir_output=output/agi_mono
      --exposure=exp_bin_agi_mono_anytime
      --origin_date=elig_date_t2dm
      --event_date=exp_date_agi_mono_anytime
      --censor_date=out_date_severecovid
      --max_fup=730
      --plot=FALSE
    needs:
    - data_process
    outputs:
      moderately_sensitive:
        output: output/agi_mono/*.csv

  km_estimates_insulin_mono:
    run: r:latest analysis/km.R 
      --df_input=output/data/data_plots.feather
      --dir_output=output/insulin_mono
      --exposure=exp_bin_insulin_mono_anytime
      --origin_date=elig_date_t2dm
      --event_date=exp_date_insulin_mono_anytime
      --censor_date=out_date_severecovid
      --max_fup=730
      --plot=FALSE
    needs:
    - data_process
    outputs:
      moderately_sensitive:
        output: output/insulin_mono/*.csv

  plot_km_estimates:
    run: r:latest analysis/km_plot.R 
    needs:
    - km_estimates_metfin
    - km_estimates_metfin_mono
    - km_estimates_dpp4_mono
    - km_estimates_tzd_mono
    - km_estimates_sglt2_mono
    - km_estimates_sulfo_mono
    - km_estimates_glp1_mono
    - km_estimates_megli_mono
    - km_estimates_agi_mono
    - km_estimates_insulin_mono
    outputs:
      moderately_sensitive:
        plot: output/data_properties/*.png

#  km_estimates_metfin:
#    run: kaplan-meier-function:v0.0.2 
#      --df_input=output/data/data_plots.feather
#      --dir_output=output/km_estimates_metfin.feather ### double-check!!!
#      --exposure=exp_bin_metfin_anytime
#      --origin_date=elig_date_t2dm
#      --event_date=exp_date_metfin_anytime
#      --censor_date=out_date_severecovid
#      --plot=TRUE
#    needs:
#    - data_process
#    outputs:
#      highly_sensitive:
#        output: output/km_estimates_metfin.feather

Timeline

Created: 8 months, 1 week ago 17 Feb 2025 09:10:55 UTC
Started: 8 months, 1 week ago 17 Feb 2025 12:45:55 UTC
Finished: 8 months, 1 week ago 17 Feb 2025 20:21:40 UTC
Runtime: 08:15:38

These timestamps are generated and stored using the UTC timezone on the TPP backend.

Job request

Status: Succeeded
Backend: TPP
Workspace: metformin-covid-main
Requested by: Alain Amstutz
Branch: main
Force run dependencies: No
Git commit hash: 62f5d8b
Requested actions: generate_dataset

data_process

km_estimates_metfin

km_estimates_metfin_mono

km_estimates_dpp4_mono

km_estimates_tzd_mono

km_estimates_sglt2_mono

km_estimates_sulfo_mono

km_estimates_glp1_mono

km_estimates_megli_mono

km_estimates_agi_mono

km_estimates_insulin_mono

plot_km_estimates

Code comparison

Compare the code used in this job request