Job request: 10012

View Repo View project.yaml

This page shows the technical details of what happened when authorised researcher Linda Nab requested one or more actions to be run against real patient data in the 96 - Covid-19 related death over time project, within a secure environment.

By cross-referencing the indicated Requested Actions with the Pipeline section below, you can infer what security level various outputs were written to. Outputs marked as highly_sensitive can never be viewed directly by a researcher; they can only request that code runs against them. Outputs marked as moderately_sensitive can be viewed by an approved researcher by logging into a highly secure environment. Only outputs marked as moderately_sensitive can be requested for release to the public, via a controlled output review service.

Jobs

ID Status Action
ne6zhhbahjkvpc6y failed calc_irs
7l4qhbf4bsiwlsv7 failed calc_irs_std
assimepeeslx626c failed calc_numbers_flowchart
2il2jhcidyxcrgbb failed calc_vax_cov
wetjr33g5ei62i5r failed create_kaplan_meier
qpuobchlhpvno3ws failed create_table_one
dn7zfno6tggchm3s failed create_table_two
kkl6day6qcoavslq failed generate_study_population_ethnicity
ljam3t7hnrhr47gp failed generate_study_population_flowchart
qt5du3oizabwo6vk failed generate_study_population_wave1
c6uf6zfgwizg7xqh failed generate_study_population_wave2
7pfd2gaxm4ii2rag failed generate_study_population_wave3
sp47rcworqcaywjy failed join_cohorts_waves
uizo6pqwbsrhow6r failed model_cox_ph
7ewnlldqrossscdj failed process_data
fhao6wggpjw2wyha failed skim_data_wave1
7att4feddvv3iomi failed skim_data_wave2
t6j5ugpop7hmypsi failed skim_data_wave3
isj6ff6avcqknzo5 failed tidy_absrisks_for_viz
pwzczip2rw4bw45d failed tidy_relrisks_for_viz
6vbrykpyi2zpjlb5 failed process_data_flowchart
h72k5z5nt2ef7spm failed skim_data_flowchart

Pipeline

Show Hide project.yaml
version: '3.0'

expectations:
  population_size: 1000

actions:

# Extract data
# When argument --index-date-range is changed, change has to be made in ./analysis/config.json too
  generate_study_population:
    run: >
      cohortextractor:latest generate_cohort 
        --study-definition study_definition 
        --skip-existing 
        --output-format=csv.gz 
        --index-date-range "2020-03-01 to 2022-02-01 by month"
    outputs:
      highly_sensitive:
        cohort: output/input_*.csv.gz

# Extract ethnicity
  generate_study_population_ethnicity:
    run: >
      cohortextractor:latest generate_cohort 
        --study-definition study_definition_ethnicity 
        --output-format=csv.gz
    outputs:
      highly_sensitive:
        cohort: output/input_ethnicity.csv.gz

# Join data
  join_cohorts:
    run: >
      cohort-joiner:v0.0.7
        --lhs output/input_202*.csv.gz
        --rhs output/input_ethnicity.csv.gz
        --output-dir=output/joined
    needs: [generate_study_population, generate_study_population_ethnicity]
    outputs:
      highly_sensitive:
        cohort: output/joined/input_202*.csv.gz

# Calculate mortality rates (crude + subgroup specific)
  calculate_measures:
    run: >
      cohortextractor:latest generate_measures 
        --study-definition study_definition
        --skip-existing
        --output-dir=output/joined
    needs: [join_cohorts]
    outputs:
      moderately_sensitive:
        measure: output/joined/measure_*_mortality_rate.csv

# Calculate mortality rates ckd_rrt subgroup
  calculate_measures_ckd_rrt:
    run: r:latest analysis/measures_calc_ckd_rrt.R
    needs: [join_cohorts]
    outputs:
      moderately_sensitive:
        measure: output/joined/measure_ckd_rrt_mortality_rate.csv

# Redact rates
  redact_rates:
    run: r:latest analysis/utils/redact_rates.R
    needs: [calculate_measures, calculate_measures_ckd_rrt]
    outputs:
      moderately_sensitive:
        csvs: output/rates/redacted/*_redacted.csv       

# Standardise crude mortality rate
  standardise_crude_rates:
    run: r:latest analysis/crude_rates_standardise.R
    needs: [redact_rates]
    outputs:
      moderately_sensitive:
        csvs: output/rates/standardised/crude_*std.csv 

# Standardise subgroup specific mortality rates
  standardise_subgroup_rates:
    run: r:latest analysis/subgroups_rates_standardise.R
    needs: [redact_rates]
    outputs:
      moderately_sensitive:
        csvs: output/rates/standardised/*_std.csv

# Process subgroup specific mortality rates
  process_subgroup_rates:
    run: r:latest analysis/utils/process_rates.R
    needs: [standardise_subgroup_rates, standardise_subgroup_rates]
    outputs:
      moderately_sensitive:
        csvs: output/rates/processed/*.csv

# Calculate standardised rate ratios
  calculate_rate_ratios:
    run: r:latest analysis/subgroups_ratios.R
    needs: [standardise_subgroup_rates, process_subgroup_rates]
    outputs:
      moderately_sensitive:
        csvs: output/ratios/*.csv

# Plot and save graphs depicting the crude rates
  visualise_crude_rates:
    run: r:latest analysis/crude_rates_visualise.R
    needs: [standardise_crude_rates]
    outputs:
      moderately_sensitive:
        pngs: output/figures/rates_crude/*.png

# Plot and save graphs depicting the subgroup specific mortality rates
  visualise_subgroup_rates:
    run: r:latest analysis/subgroups_rates_visualise.R
    needs: [standardise_subgroup_rates, process_subgroup_rates]
    outputs:
      moderately_sensitive:
        pngs: output/figures/rates_subgroups/*.png

# Plot and save graphs depicting the subgroup specific mortality ratios
  visualise_subgroup_ratios:
    run: r:latest analysis/subgroups_ratios_visualise.R
    needs: [calculate_rate_ratios]
    outputs:
      moderately_sensitive:
        pngs: output/figures/ratios_subgroups/*.png

# SECOND PART OF STUDY
  generate_study_population_flowchart:
    run: >
      cohortextractor:latest generate_cohort 
        --study-definition study_definition_flowchart
        --skip-existing 
        --output-format=csv.gz
    outputs:
      highly_sensitive:
        cohort: output/input_flowchart.csv.gz

  # Process data flowchart
  process_data_flowchart:
    run: r:latest analysis/data_flowchart_process.R
    needs: [generate_study_population_flowchart]
    outputs:
      highly_sensitive: 
        rds: output/processed/input_flowchart.rds
  
  # Skim data flowchart
  skim_data_flowchart:
    run: r:latest analysis/data_skim.R output/processed/input_flowchart.rds output/data_properties
    needs: [process_data_flowchart]
    outputs: 
      moderately_sensitive:
        txt1: output/data_properties/input_flowchart_skim.txt
        txt2: output/data_properties/input_flowchart_coltypes.txt
        txt3: output/data_properties/input_flowchart_tabulate.txt

  # Numbers for flowchart
  calc_numbers_flowchart:
    run: r:latest analysis/flowchart.R
    needs: [process_data_flowchart]
    outputs:
      moderately_sensitive:
        cohort: output/tables/flowchart/wave1_flowchart.csv

  generate_study_population_wave1:
    run: >
      cohortextractor:latest generate_cohort 
        --study-definition study_definition_wave1 
        --skip-existing 
        --output-format=csv.gz
    outputs:
      highly_sensitive:
        cohort: output/input_wave1.csv.gz

  generate_study_population_wave2:
    run: >
      cohortextractor:latest generate_cohort 
        --study-definition study_definition_wave2
        --skip-existing 
        --output-format=csv.gz
    outputs:
      highly_sensitive:
        cohort: output/input_wave2.csv.gz

  generate_study_population_wave3:
    run: >
      cohortextractor:latest generate_cohort 
        --study-definition study_definition_wave3
        --skip-existing 
        --output-format=csv.gz
    outputs:
      highly_sensitive:
        cohort: output/input_wave3.csv.gz

# Join data
  join_cohorts_waves:
    run: >
      cohort-joiner:v0.0.7
        --lhs output/input_wave*.csv.gz
        --rhs output/input_ethnicity.csv.gz
        --output-dir=output/joined
    needs: [generate_study_population_wave1, generate_study_population_wave2, generate_study_population_wave3, generate_study_population_ethnicity]
    outputs:
      highly_sensitive:
        cohort: output/joined/input_wave*.csv.gz

# Process data
  process_data:
    run: r:latest analysis/data_process.R
    needs: [join_cohorts_waves]
    outputs:
      highly_sensitive: 
        rds: output/processed/input_wave*.rds

# Skim data
  skim_data_wave1:
    run: r:latest analysis/data_skim.R output/processed/input_wave1.rds output/data_properties
    needs: [process_data]
    outputs: 
      moderately_sensitive:
        txt1: output/data_properties/input_wave1_skim.txt
        txt2: output/data_properties/input_wave1_coltypes.txt
        txt3: output/data_properties/input_wave1_tabulate.txt

  skim_data_wave2:
    run: r:latest analysis/data_skim.R output/processed/input_wave2.rds output/data_properties
    needs: [process_data]
    outputs: 
      moderately_sensitive:
        txt1: output/data_properties/input_wave2_skim.txt
        txt2: output/data_properties/input_wave2_coltypes.txt
        txt3: output/data_properties/input_wave2_tabulate.txt

  skim_data_wave3:
    run: r:latest analysis/data_skim.R output/processed/input_wave3.rds output/data_properties
    needs: [process_data]
    outputs: 
      moderately_sensitive:
        txt1: output/data_properties/input_wave3_skim.txt
        txt2: output/data_properties/input_wave3_coltypes.txt
        txt3: output/data_properties/input_wave3_tabulate.txt

# Vaccine coverage
  calc_vax_cov:
    run: r:latest analysis/vaccine_coverage_calc.R
    needs: [process_data]
    outputs: 
      moderately_sensitive:
        csvs: output/tables/wave*_vax_coverage.csv

# Incidence rates (crude)
  calc_irs:
    run: r:latest analysis/waves_irs.R
    needs: [process_data]
    outputs: 
      moderately_sensitive:
        csvs: output/tables/wave*_ir.csv
        csv: output/tables/ir_crude.csv

# Create table one
  create_table_one:
    run: r:latest analysis/table_one.R
    needs: [process_data, calc_irs]
    outputs: 
      moderately_sensitive:
        csv: output/tables/table1.csv

# Incidence rates (crude)
  calc_irs_std:
    run: r:latest analysis/waves_std_irs.R
    needs: [process_data]
    outputs: 
      moderately_sensitive:
        csvs: output/tables/wave*_ir_std.csv

# Kaplan-Meier
  create_kaplan_meier:
    run: r:latest analysis/waves_kaplan_meier.R
    needs: [process_data]
    outputs:
      moderately_sensitive:
        pngs: output/figures/kaplan_meier/wave*_*.png

# COX ph models
  model_cox_ph:
    run: r:latest analysis/waves_model_survival.R
    needs: [process_data]
    outputs: 
      moderately_sensitive:
        csvs1: output/tables/wave*_effect_estimates.csv
        csvs2: output/tables/wave*_ph_tests.csv
        csvs3: output/tables/wave*_log_file.csv

# Create table two
  create_table_two:
    run: r:latest analysis/table_two.R
    needs: [model_cox_ph]
    outputs:
      moderately_sensitive: 
        html: output/tables/table2.html
        csv: output/tables/table2.csv

# Tidy absrisks (IRs) for viz
  tidy_absrisks_for_viz:
    run: r:latest analysis/absrisks_tidy_for_viz.R
    needs: [calc_irs, calc_irs_std, calc_vax_cov]
    outputs:
      moderately_sensitive: 
        csv: output/tables/absrisks_for_viz_tidied.csv

# Tidy relrisks (HRs) for viz
  tidy_relrisks_for_viz:
    run: r:latest analysis/relrisks_tidy_for_viz.R
    needs: [model_cox_ph, calc_vax_cov]
    outputs:
      moderately_sensitive: 
        csv: output/tables/relrisks_for_viz_tidied.csv
        
# Wave-specific HR plots
#  plot_relrisks:
#    run: r:latest -e 'rmarkdown::render("analysis/wave_HRs.Rmd", knit_root_dir = "/workspace", output_dir="/workspace/output/markdown")'
#    needs: [tidy_relrisks_for_viz]
#    outputs:
#      moderately_sensitive:
#        csv: output/markdown/wave_HRs.html

State

State is inferred from the related Jobs.

Status: Failed

Timings

Timings set to UTC timezone.

  • Created:
  • Started:
  • Finished:
  • Runtime: 01:26:51

Config

  • Backend:
    TPP
  • Workspace:
    covid_mortality_over_time
  • Branch:
    main
  • Creator:
    LindaNab
  • Force run dependencies:
    False
  • Git Commit Hash:
    308d854
  • Requested actions:
    • generate_study_population_ethnicity
    • generate_study_population_flowchart
    • process_data_flowchart
    • skim_data_flowchart
    • calc_numbers_flowchart
    • generate_study_population_wave1
    • generate_study_population_wave2
    • generate_study_population_wave3
    • join_cohorts_waves
    • process_data
    • skim_data_wave1
    • skim_data_wave2
    • skim_data_wave3
    • calc_vax_cov
    • calc_irs
    • create_table_one
    • calc_irs_std
    • create_kaplan_meier
    • model_cox_ph
    • create_table_two
    • tidy_absrisks_for_viz
    • tidy_relrisks_for_viz