Job request: 19530

Organisation:: Bennett Institute
Workspace:: pax-usernonuser
ID:: npxmnsqizj7uau3e

This page shows the technical details of what happened when the authorised researcher Linda Nab requested one or more actions to be run against real patient data within a secure environment.

By cross-referencing the list of jobs with the pipeline section below, you can infer what security level the outputs were written to.

The output security levels are:

highly_sensitive
- Researchers can never directly view these outputs
- Researchers can only request code is run against them
moderately_sensitive
- Can be viewed by an approved researcher by logging into a highly secure environment
- These are the only outputs that can be requested for public release via a controlled output review service.

Jobs

Action:

generate_study_population

Status:

Status: Succeeded

Job identifier:

5muzimygfyfn2fmu
Action:

generate_study_population_flowchart

Status:

Status: Succeeded

Job identifier:

eslwigdlibfztmrp
Action:

data_process_flowchart

Status:

Status: Succeeded

Job identifier:

mjzv6hxeq6nnprvr
Action:

generate_study_population_pax_trt

Status:

Status: Succeeded

Job identifier:

wnodfxu2cd6d2eab
Action:

data_process

Status:

Status: Succeeded

Job identifier:

5svn43awnrsoqzpq
Action:

create_flowchart

Status:

Status: Succeeded

Job identifier:

xtduwqro2ksgzjsk
Action:

create_table1

Status:

Status: Succeeded

Job identifier:

mxegmiarihympeeb
Action:

data_prepare_seq_trials

Status:

Status: Succeeded

Job identifier:

uivsws6qbcyhxuvp
Action:

data_properties

Status:

Status: Succeeded

Job identifier:

jtmxlqmsj33utgus
Action:

describe_distr_trt_init

Status:

Status: Succeeded

Job identifier:

glaqtq66gphl5lma
Action:

describe_outcomes_in_trt_window

Status:

Status: Succeeded

Job identifier:

fzen3sdr776zecyh
Action:

describe_postest_pop

Status:

Status: Succeeded

Job identifier:

rgzsvszqy7uvdawg
Action:

describe_prop_trt

Status:

Status: Succeeded

Job identifier:

6ckvns7l7ws4rxrw
Action:

describe_size_trials

Status:

Status: Succeeded

Job identifier:

5dzxn52b5jyzfcah
Action:

describe_trt_contra

Status:

Status: Succeeded

Job identifier:

kvgd2luih26ix3ur
Action:

describe_trt_preds_all_ci

Status:

Status: Succeeded

Job identifier:

u6dyoexslpx4td2w
Action:

describe_trt_preds_excl_drugs_dnu

Status:

Status: Succeeded

Job identifier:

jq476z3w43lylwij

Pipeline

Show project.yaml

version: '3.0'

expectations:
  population_size: 5000

actions:

  ## # # # # # # # # # # # # # # # # # # # 
  ## # # # # # # # # # # # # # # # # # # # 
  ## Data extraction 
  ## # # # # # # # # # # # # # # # # # # # 
  ## # # # # # # # # # # # # # # # # # # # 

  generate_study_population:
    run: cohortextractor:latest generate_cohort --study-definition study_definition --output-format=csv.gz
    outputs:
      highly_sensitive:
        cohort: output/input.csv.gz

  generate_study_population_flowchart:
    run: cohortextractor:latest generate_cohort --study-definition study_definition_flowchart --output-format=csv.gz
    outputs:
      highly_sensitive:
        cohort: output/input_flowchart.csv.gz

  generate_study_population_pax_trt:
    run: cohortextractor:latest generate_cohort --study-definition study_definition_pax_trt --output-format=csv.gz
    outputs:
      highly_sensitive:
        cohort: output/input_pax_trt.csv.gz

  ## # # # # # # # # # # # # # # # # # # # 
  ## # # # # # # # # # # # # # # # # # # # 
  ## Data cleaning and description
  ## # # # # # # # # # # # # # # # # # # # 
  ## # # # # # # # # # # # # # # # # # # # 

  data_process:
    run: r:latest analysis/data_process.R
    needs: [generate_study_population]
    outputs:
      highly_sensitive:
        data1: output/data/data_processed.rds
        data2: output/data/data_processed_grace6.rds
        data3: output/data/data_processed_grace7.rds
        data4: output/data/data_processed_grace8.rds
        data1_excl: output/data/data_processed_excl_contraindicated.rds
        data2_excl: output/data/data_processed_excl_contraindicated_grace6.rds
        data3_excl: output/data/data_processed_excl_contraindicated_grace7.rds
        data4_excl: output/data/data_processed_excl_contraindicated_grace8.rds
        rds1: output/data_properties/n_excluded.rds
        rds2: output/data_properties/n_excluded_contraindicated.rds

  data_process_flowchart:
    run: r:latest analysis/data_process_flowchart.R
    needs: [generate_study_population_flowchart]
    outputs:
      highly_sensitive:
        data: output/data/data_flowchart_processed.rds
  
  data_properties:
    run: r:latest analysis/data_properties/data_properties.R output/data/data_processed.rds output/data_properties
    needs: [data_process]
    outputs:
      moderately_sensitive:
        txt1: output/data_properties/data_processed_skim.txt
        txt2: output/data_properties/data_processed_coltypes.txt
        txt3: output/data_properties/data_processed_tabulate.txt

  create_flowchart:
    run: r:latest analysis/flowchart.R
    needs: [data_process_flowchart, data_process]
    outputs:
      moderately_sensitive:
        csv1: output/flowchart/flowchart.csv
        csv2: output/flowchart/flowchart_redacted.csv

  ## # # # # # # # # # # # # # # # # # # # 
  ## # # # # # # # # # # # # # # # # # # # 
  ## Descriptive analyses
  ## # # # # # # # # # # # # # # # # # # # 
  ## # # # # # # # # # # # # # # # # # # #

  describe_distr_trt_init:
    run: r:latest analysis/descriptives/describe_distr_trt_init.R
    needs: [data_process]
    outputs:
      moderately_sensitive:
        csv1: output/descriptives/distr_pax_init.csv
        csv2: output/descriptives/distr_pax_init_red.csv

  describe_outcomes_in_trt_window:
    run: r:latest analysis/descriptives/describe_outcomes_in_trt_window.R
    needs: [data_process]
    outputs:
      moderately_sensitive:
        csv1: output/descriptives/outcomes_in_trt_window.csv
        csv2: output/descriptives/outcomes_in_trt_window_red.csv
        csv3: output/descriptives/outcomes_day0.csv
  
  describe_prop_trt:
    run: r:latest analysis/descriptives/describe_prop_trt.R
    needs: [data_process]
    outputs:
      moderately_sensitive:
        csv1: output/descriptives/prop_trt.csv
        csv2: output/descriptives/prop_trt_red.csv

  describe_trt_contra:
    run: r:latest analysis/descriptives/describe_trt_contra.R
    needs: [data_process]
    outputs:
      moderately_sensitive:
        csv1: output/descriptives/trt_contra.csv
        csv2: output/descriptives/trt_contra_red.csv
        csvs1: output/descriptives/n_codes_*.csv

  describe_trt_preds_all_ci:
    run: r:latest analysis/descriptives/describe_trt_preds.R all_ci
    needs: [data_process]
    outputs:
      highly_sensitive:
        data1: output/descriptives/psModel_all_ci.rds     
      moderately_sensitive:
        png1: output/descriptives/psOverlap_untrimmed_all_ci.png
        png2: output/descriptives/psOverlap_trimmed_all_ci.png
        png3: output/descriptives/psCoefs_all_ci.png
        csv1: output/descriptives/trimming_descriptives_all_ci.csv

  describe_trt_preds_excl_drugs_dnu:
    run: r:latest analysis/descriptives/describe_trt_preds.R ci_drugs_dnu
    needs: [data_process]
    outputs:
      highly_sensitive:
        data1: output/descriptives/psModel_excl_drugs_dnu.rds
      moderately_sensitive:
        png1: output/descriptives/psOverlap_untrimmed_excl_drugs_dnu.png
        png2: output/descriptives/psOverlap_trimmed_excl_drugs_dnu.png
        png3: output/descriptives/psCoefs_excl_drugs_dnu.png
        csv1: output/descriptives/trimming_descriptives_excl_drugs_dnu.csv
        
  create_table1:
    run: r:latest analysis/descriptives/table1.R
    needs: [data_process]
    outputs:
      moderately_sensitive:
        csv1: output/tables/table1.csv
        csv2: output/tables/table1_red.csv
  
  describe_postest_pop:
    run: r:latest analysis/descriptives/describe_postest_pop.R
    needs: [data_process]
    outputs:
      moderately_sensitive:
        csv1: output/descriptives/postest_pop.csv
        csv2: output/descriptives/postest_pop_red.csv
    
  ## # # # # # # # # # # # # # # # # # # # 
  ## # # # # # # # # # # # # # # # # # # # 
  ## Sequential trials
  ## # # # # # # # # # # # # # # # # # # # 
  ## # # # # # # # # # # # # # # # # # # #
  data_prepare_seq_trials:
    run: r:latest analysis/seq_trials/prepare_data.R
    needs: [data_process]
    outputs:
      highly_sensitive:
        data1: output/data/data_seq_trials_monthly.feather
        data2: output/data/data_seq_trials_weekly.feather

  describe_size_trials:
    run: r:latest analysis/seq_trials/descriptives/describe_size_trials.R
    needs: [data_prepare_seq_trials]
    outputs:
      moderately_sensitive:
        csv1: output/seq_trials/descriptives/data_flow_seq_trials_monthly.csv
        csv2: output/seq_trials/descriptives/data_flow_seq_trials_weekly.csv
        csv3: output/seq_trials/descriptives/data_flow_seq_trials_monthly_red.csv
        csv4: output/seq_trials/descriptives/data_flow_seq_trials_weekly_red.csv

Timeline

Created: 2 years, 3 months ago 04 Aug 2023 13:29:17 UTC
Started: 2 years, 3 months ago 04 Aug 2023 15:17:21 UTC
Finished: 2 years, 2 months ago 05 Aug 2023 07:19:00 UTC
Runtime: 22:19:32

These timestamps are generated and stored using the UTC timezone on the TPP backend.

Job request

Status: Succeeded
Backend: TPP
Workspace: pax-usernonuser
Requested by: Linda Nab
Branch: main
Force run dependencies: No
Git commit hash: 141b12d
Requested actions: generate_study_population

generate_study_population_flowchart

generate_study_population_pax_trt

data_process

data_process_flowchart

data_properties

create_flowchart

describe_distr_trt_init

describe_outcomes_in_trt_window

describe_prop_trt

describe_trt_contra

describe_trt_preds_all_ci

describe_trt_preds_excl_drugs_dnu

create_table1

describe_postest_pop

data_prepare_seq_trials

describe_size_trials

Code comparison

Compare the code used in this job request