Job request: 9192

View Repo View project.yaml

This page shows the technical details of what happened when authorised researcher Linda Nab requested one or more actions to be run against real patient data in the 96 - Covid-19 related death over time project, within a secure environment.

By cross-referencing the indicated Requested Actions with the Pipeline section below, you can infer what security level various outputs were written to. Outputs marked as highly_sensitive can never be viewed directly by a researcher; they can only request that code runs against them. Outputs marked as moderately_sensitive can be viewed by an approved researcher by logging into a highly secure environment. Only outputs marked as moderately_sensitive can be requested for release to the public, via a controlled output review service.

Jobs

ID Status Action
5me5ctalxiynipzl succeeded calculate_measures
mu2yr4ipswvmdxxr succeeded calculate_measures_ckd_rrt
omtjaqt3v4rji5dh succeeded calculate_rate_ratios
glsap27z5v43imjx succeeded create_kaplan_meier
cthp3aomlegorlrx succeeded create_table_one
jcdhrrm7fbnpgsm2 succeeded create_table_two
gmqfihrxadqaqgbn succeeded generate_study_population
3znlefbchh4bvngl succeeded generate_study_population_ethnicity
lr2zd6bcckctu66w succeeded generate_study_population_wave1
ucwkhdigrki57sbn succeeded generate_study_population_wave2
xbfzfjklulppdr2y succeeded generate_study_population_wave3
6ugm2hpw47w3jd5d succeeded join_cohorts
vccr6cw65hw36u65 succeeded join_cohorts_waves
epn7agrry3jupeff succeeded model_cox_ph
kdg5rnfbtjtn2b2d succeeded process_data
afjzgmaa6szpdrv6 succeeded process_subgroup_rates
ccelznqeukjteran succeeded redact_rates
dxkcnme5jhnypsvz succeeded skim_data_wave1
eofgjuhtdmibh52u succeeded skim_data_wave2
y53porqunxa3msfb succeeded skim_data_wave3
3tv3d4wwnctop6mv succeeded standardise_crude_rates
2pzczb4lvxv7w43e succeeded standardise_subgroup_rates
smawictpsxaquaym succeeded visualise_crude_rates
uar56epeteqd5v5v succeeded visualise_subgroup_rates
2tw6znbgwfqg7icp succeeded visualise_subgroup_ratios
pe24anu53kkhjugi succeeded calc_vax_cov
ti4ampt7v7hgli6a succeeded tidy_relrisks_for_viz

Pipeline

Show Hide project.yaml
version: '3.0'

expectations:
  population_size: 1000

actions:

# Extract data
# When argument --index-date-range is changed, change has to be made in ./analysis/config.json too
  generate_study_population:
    run: >
      cohortextractor:latest generate_cohort 
        --study-definition study_definition 
        --skip-existing 
        --output-format=csv.gz 
        --index-date-range "2020-03-01 to 2022-02-01 by month"
    outputs:
      highly_sensitive:
        cohort: output/input_*.csv.gz

# Extract ethnicity
  generate_study_population_ethnicity:
    run: >
      cohortextractor:latest generate_cohort 
        --study-definition study_definition_ethnicity 
        --output-format=csv.gz
    outputs:
      highly_sensitive:
        cohort: output/input_ethnicity.csv.gz

# Join data
  join_cohorts:
    run: >
      cohort-joiner:v0.0.7
        --lhs output/input_202*.csv.gz
        --rhs output/input_ethnicity.csv.gz
        --output-dir=output/joined
    needs: [generate_study_population, generate_study_population_ethnicity]
    outputs:
      highly_sensitive:
        cohort: output/joined/input_202*.csv.gz

# Calculate mortality rates (crude + subgroup specific)
  calculate_measures:
    run: >
      cohortextractor:latest generate_measures 
        --study-definition study_definition
        --skip-existing
        --output-dir=output/joined
    needs: [join_cohorts]
    outputs:
      moderately_sensitive:
        measure: output/joined/measure_*_mortality_rate.csv

# Calculate mortality rates ckd_rrt subgroup
  calculate_measures_ckd_rrt:
    run: r:latest analysis/measures_calc_ckd_rrt.R
    needs: [join_cohorts]
    outputs:
      moderately_sensitive:
        measure: output/joined/measure_ckd_rrt_mortality_rate.csv

# Redact rates
  redact_rates:
    run: r:latest analysis/utils/redact_rates.R
    needs: [calculate_measures, calculate_measures_ckd_rrt]
    outputs:
      moderately_sensitive:
        csvs: output/rates/redacted/*_redacted.csv       

# Standardise crude mortality rate
  standardise_crude_rates:
    run: r:latest analysis/crude_rates_standardise.R
    needs: [redact_rates]
    outputs:
      moderately_sensitive:
        csvs: output/rates/standardised/crude_*std.csv 

# Standardise subgroup specific mortality rates
  standardise_subgroup_rates:
    run: r:latest analysis/subgroups_rates_standardise.R
    needs: [redact_rates]
    outputs:
      moderately_sensitive:
        csvs: output/rates/standardised/*_std.csv

# Process subgroup specific mortality rates
  process_subgroup_rates:
    run: r:latest analysis/utils/process_rates.R
    needs: [standardise_subgroup_rates, standardise_subgroup_rates]
    outputs:
      moderately_sensitive:
        csvs: output/rates/processed/*.csv

# Calculate standardised rate ratios
  calculate_rate_ratios:
    run: r:latest analysis/subgroups_ratios.R
    needs: [standardise_subgroup_rates, process_subgroup_rates]
    outputs:
      moderately_sensitive:
        csvs: output/ratios/*.csv

# Plot and save graphs depicting the crude rates
  visualise_crude_rates:
    run: r:latest analysis/crude_rates_visualise.R
    needs: [standardise_crude_rates]
    outputs:
      moderately_sensitive:
        pngs: output/figures/rates_crude/*.png

# Plot and save graphs depicting the subgroup specific mortality rates
  visualise_subgroup_rates:
    run: r:latest analysis/subgroups_rates_visualise.R
    needs: [standardise_subgroup_rates, process_subgroup_rates]
    outputs:
      moderately_sensitive:
        pngs: output/figures/rates_subgroups/*.png

# Plot and save graphs depicting the subgroup specific mortality ratios
  visualise_subgroup_ratios:
    run: r:latest analysis/subgroups_ratios_visualise.R
    needs: [calculate_rate_ratios]
    outputs:
      moderately_sensitive:
        pngs: output/figures/ratios_subgroups/*.png

# SECOND PART OF STUDY
  generate_study_population_wave1:
    run: >
      cohortextractor:latest generate_cohort 
        --study-definition study_definition_wave1 
        --skip-existing 
        --output-format=csv.gz
    outputs:
      highly_sensitive:
        cohort: output/input_wave1.csv.gz

  generate_study_population_wave2:
    run: >
      cohortextractor:latest generate_cohort 
        --study-definition study_definition_wave2
        --skip-existing 
        --output-format=csv.gz
    outputs:
      highly_sensitive:
        cohort: output/input_wave2.csv.gz

  generate_study_population_wave3:
    run: >
      cohortextractor:latest generate_cohort 
        --study-definition study_definition_wave3
        --skip-existing 
        --output-format=csv.gz
    outputs:
      highly_sensitive:
        cohort: output/input_wave3.csv.gz

# Join data
  join_cohorts_waves:
    run: >
      cohort-joiner:v0.0.7
        --lhs output/input_wave*.csv.gz
        --rhs output/input_ethnicity.csv.gz
        --output-dir=output/joined
    needs: [generate_study_population_wave1, generate_study_population_wave2, generate_study_population_wave3, generate_study_population_ethnicity]
    outputs:
      highly_sensitive:
        cohort: output/joined/input_wave*.csv.gz

# Process data
  process_data:
    run: r:latest analysis/data_process.R
    needs: [join_cohorts_waves]
    outputs:
      highly_sensitive: 
        rds: output/processed/input_wave*.rds

# Skim data
  skim_data_wave1:
    run: r:latest analysis/data_skim.R output/processed/input_wave1.rds output/data_properties
    needs: [process_data]
    outputs: 
      moderately_sensitive:
        txt1: output/data_properties/input_wave1_skim.txt
        txt2: output/data_properties/input_wave1_coltypes.txt
        txt3: output/data_properties/input_wave1_tabulate.txt

  skim_data_wave2:
    run: r:latest analysis/data_skim.R output/processed/input_wave2.rds output/data_properties
    needs: [process_data]
    outputs: 
      moderately_sensitive:
        txt1: output/data_properties/input_wave2_skim.txt
        txt2: output/data_properties/input_wave2_coltypes.txt
        txt3: output/data_properties/input_wave2_tabulate.txt

  skim_data_wave3:
    run: r:latest analysis/data_skim.R output/processed/input_wave3.rds output/data_properties
    needs: [process_data]
    outputs: 
      moderately_sensitive:
        txt1: output/data_properties/input_wave3_skim.txt
        txt2: output/data_properties/input_wave3_coltypes.txt
        txt3: output/data_properties/input_wave3_tabulate.txt

# Create table one
  create_table_one:
    run: r:latest analysis/table_one.R
    needs: [process_data]
    outputs:
      moderately_sensitive: 
        html: output/tables/table1.html

# Vaccine coverage
  calc_vax_cov:
    run: r:latest analysis/vaccine_coverage_calc.R
    needs: [process_data]
    outputs: 
      moderately_sensitive:
        csvs: output/tables/wave*_vax_coverage.csv

# Kaplan-Meier
  create_kaplan_meier:
    run: r:latest analysis/waves_kaplan_meier.R
    needs: [process_data]
    outputs:
      moderately_sensitive:
        pngs: output/figures/kaplan_meier/wave*_*.png

# COX ph models
  model_cox_ph:
    run: r:latest analysis/waves_model_survival.R
    needs: [process_data]
    outputs: 
      moderately_sensitive:
        csvs1: output/tables/wave*_effect_estimates.csv
        csvs2: output/tables/wave*_ph_tests.csv
        csvs3: output/tables/wave*_log_file.csv

# Create table two
  create_table_two:
    run: r:latest analysis/table_two.R
    needs: [model_cox_ph]
    outputs:
      moderately_sensitive: 
        html: output/tables/table2.html

# Tidy relrisks (HRs) for viz
  tidy_relrisks_for_viz:
    run: r:latest analysis/relrisks_tidy_for_viz.R
    needs: [model_cox_ph, calc_vax_cov]
    outputs:
      moderately_sensitive: 
        csv: output/tables/relrisks_for_viz_tidied.csv

State

State is inferred from the related Jobs.

Status: Succeeded

Timings

Timings set to UTC timezone.

  • Created:
  • Started:
  • Finished:
  • Runtime: 151:43:52

Config