Job request: 8980

View Repo View project.yaml

This page shows the technical details of what happened when authorised researcher Yinghui Wei requested one or more actions to be run against real patient data in the long-covid-risk-factors project, within a secure environment.

By cross-referencing the indicated Requested Actions with the Pipeline section below, you can infer what security level various outputs were written to. Outputs marked as highly_sensitive can never be viewed directly by a researcher; they can only request that code runs against them. Outputs marked as moderately_sensitive can be viewed by an approved researcher by logging into a highly secure environment. Only outputs marked as moderately_sensitive can be requested for release to the public, via a controlled output review service.

Jobs

ID Status Action
yqe5twmjyzmuz4k6 failed development_cox_model_all
4sqofhfscyfeykue failed development_cox_model_all_vax_td
ah7miwd2c2ripzzq failed development_cox_model_vax_c
tpvl3ujakn2qjxl7 failed evaluation_cox_model_all
wky2b4q577jpjh6g failed evaluation_cox_model_all_vax_td
rt7hcn6km4wy5wdc failed evaluation_cox_model_vax_c
bk5zbhmjob35koyq failed validation_cox_model_all
fx5h5ljn2k3pej6l failed validation_cox_model_all_vax_td
frb3g6m34xgm3yly failed validation_cox_model_vax_c
46ymltfwsj3got2k succeeded stage0_data_cleaning_all
sahi6d45tzpaetn5 failed stage1_define_eligible_population_all
mkeu3hqkl76ecsqg failed table_1_all
27r3hhp2g6npc5je failed table_2_all
w2vom2ni6kriu5tg failed table_3_all
l6peszlr2uoratue failed figure_1_all
iylu4w2aslnswwir failed figure_2_all
uuzgvivg43hc22w6 failed figure_cum_prob_km_all
6liun2e4z5rtkxxc failed suppl_table_1_all
5qwqxlfdokkyibak failed suppl_figure_1_all
txmsal5sbk2duxpe failed summarise_survival_data_all

Pipeline

Show Hide project.yaml
version: '3.0'

expectations:

  population_size: 15000

actions:

  ## # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # 
  ## DO NOT EDIT project.yaml DIRECTLY 
  ## This file is created by create_project_actions.R 
  ## Edit and run create_project_actions.R to update the project.yaml 
  ## # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # 
  ##  
  ## Part 1. Generate Study Population 
  ##  
  ## Generate dummy data for all eligible population 

  generate_study_population_all:
    run: cohortextractor:latest generate_cohort --study-definition study_definition_all
      --output-format feather
    outputs:
      highly_sensitive:
        cohort: output/input_all.feather

  ## Generate dummy data for study population - vaccinated 

  generate_study_population_vaccinated:
    run: cohortextractor:latest generate_cohort --study-definition study_definition_vaccinated
      --output-format feather
    outputs:
      highly_sensitive:
        cohort: output/input_vaccinated.feather

  ## Generate dummy data for study population - infected 

  generate_study_population_infected:
    run: cohortextractor:latest generate_cohort --study-definition study_definition_infected
      --output-format feather
    outputs:
      highly_sensitive:
        cohort: output/input_infected.feather

  ## Part 2. Create Analysis Data Sets 
  ## Data Cleaning 
  ## Stage 0 Data Cleaning - all 

  stage0_data_cleaning_all:
    run: r:latest analysis/stage0_data_cleaning.R all
    needs:
    - generate_study_population_all
    outputs:
      highly_sensitive:
        cohort: output/input_stage0_all.rds
      moderately_sensitive:
        variable_check_table_CSV: output/not_for_review/descriptives/table_0_all.csv
        variable_check_table_HTML: output/not_for_review/descriptives/table_0_all.html
        histogram_numerical_variable: output/not_for_review/descriptives/histogram_*_all.svg

  ## Stage 0 Data Cleaning - vaccinated 

  stage0_data_cleaning_vaccinated:
    run: r:latest analysis/stage0_data_cleaning.R vaccinated
    needs:
    - generate_study_population_vaccinated
    outputs:
      highly_sensitive:
        cohort: output/input_stage0_vaccinated.rds
      moderately_sensitive:
        variable_check_table_CSV: output/not_for_review/descriptives/table_0_vaccinated.csv
        variable_check_table_HTML: output/not_for_review/descriptives/table_0_vaccinated.html
        histogram_numerical_variable: output/not_for_review/descriptives/histogram_*_vaccinated.svg

  ## Stage 0 Data Cleaning - infected 

  stage0_data_cleaning_infected:
    run: r:latest analysis/stage0_data_cleaning.R infected
    needs:
    - generate_study_population_infected
    outputs:
      highly_sensitive:
        cohort: output/input_stage0_infected.rds
      moderately_sensitive:
        variable_check_table_CSV: output/not_for_review/descriptives/table_0_infected.csv
        variable_check_table_HTML: output/not_for_review/descriptives/table_0_infected.html
        histogram_numerical_variable: output/not_for_review/descriptives/histogram_*_infected.svg

  ## Define eligible population 
  ## Stage 1 define eligible population - all 

  stage1_define_eligible_population_all:
    run: r:latest analysis/stage1_define_eligible_population.R all
    needs:
    - stage0_data_cleaning_all
    outputs:
      highly_sensitive:
        cohort: output/input_stage1_all.rds
      moderately_sensitive:
        flow_chart_csv: output/flow_chart_all.csv
        flow_chart_html: output/flow_chart_all.html

  ## Stage 1 define eligible population - vaccinated 

  stage1_define_eligible_population_vaccinated:
    run: r:latest analysis/stage1_define_eligible_population.R vaccinated
    needs:
    - stage0_data_cleaning_vaccinated
    outputs:
      highly_sensitive:
        cohort: output/input_stage1_vaccinated.rds
      moderately_sensitive:
        flow_chart_csv: output/flow_chart_vaccinated.csv
        flow_chart_html: output/flow_chart_vaccinated.html

  ## Stage 1 define eligible population - infected 

  stage1_define_eligible_population_infected:
    run: r:latest analysis/stage1_define_eligible_population.R infected
    needs:
    - stage0_data_cleaning_infected
    outputs:
      highly_sensitive:
        cohort: output/input_stage1_infected.rds
      moderately_sensitive:
        flow_chart_csv: output/flow_chart_infected.csv
        flow_chart_html: output/flow_chart_infected.html

  ## Part 3. Figures and tables 
  ## Table 1. Patient Characteristics - all 

  table_1_all:
    run: r:latest analysis/table_1.R all
    needs:
    - stage1_define_eligible_population_all
    outputs:
      moderately_sensitive:
        descriptive_table_CSV: output/review/descriptives/table_1_all.csv
        descriptive_table_HTML: output/review/descriptives/table_1_all.html

  ## Table 1. Patient Characteristics - vaccinated 

  table_1_vaccinated:
    run: r:latest analysis/table_1.R vaccinated
    needs:
    - stage1_define_eligible_population_vaccinated
    outputs:
      moderately_sensitive:
        descriptive_table_CSV: output/review/descriptives/table_1_vaccinated.csv
        descriptive_table_HTML: output/review/descriptives/table_1_vaccinated.html

  ## Table 1. Patient Characteristics - infected 

  table_1_infected:
    run: r:latest analysis/table_1.R infected
    needs:
    - stage1_define_eligible_population_infected
    outputs:
      moderately_sensitive:
        descriptive_table_CSV: output/review/descriptives/table_1_infected.csv
        descriptive_table_HTML: output/review/descriptives/table_1_infected.html

  ## Define eligible population 
  ## Table 2. Event count and incidence rate - all 

  table_2_all:
    run: r:latest analysis/table_2.R all
    needs:
    - stage1_define_eligible_population_all
    outputs:
      moderately_sensitive:
        incidence_rate_table_CSV: output/review/descriptives/table_2_all.csv
        incidence_rate_talbe_HTML: output/review/descriptives/table_2_all.html

  ## Table 2. Event count and incidence rate - vaccinated 

  table_2_vaccinated:
    run: r:latest analysis/table_2.R vaccinated
    needs:
    - stage1_define_eligible_population_vaccinated
    outputs:
      moderately_sensitive:
        incidence_rate_table_CSV: output/review/descriptives/table_2_vaccinated.csv
        incidence_rate_talbe_HTML: output/review/descriptives/table_2_vaccinated.html

  ## Table 2. Event count and incidence rate - infected 

  table_2_infected:
    run: r:latest analysis/table_2.R infected
    needs:
    - stage1_define_eligible_population_infected
    outputs:
      moderately_sensitive:
        incidence_rate_table_CSV: output/review/descriptives/table_2_infected.csv
        incidence_rate_talbe_HTML: output/review/descriptives/table_2_infected.html

  ## table_3 - sequence count 

  table_3_all:
    run: r:latest analysis/table_3.R
    needs:
    - stage1_define_eligible_population_all
    outputs:
      moderately_sensitive:
        sequence_count_table_CSV: output/review/descriptives/table_3.csv
        sequence_count_table_HTML: output/review/descriptives/table_3.html

  ## Figure_1 - long covid count 

  figure_1_all:
    run: r:latest analysis/figure_1.R
    needs:
    - stage1_define_eligible_population_all
    outputs:
      moderately_sensitive:
        figure_long_covid_count_all: output/figure_1_*.svg
        table_csv_long_covid_count_all: output/review/descriptives/long_covid_count_*_all.csv
        table_html_long_covid_count_all: output/review/descriptives/long_covid_count_*_all.html

  ## Figure_2 - days from covid to long covid 

  figure_2_all:
    run: r:latest analysis/figure_2.R
    needs:
    - stage1_define_eligible_population_all
    outputs:
      moderately_sensitive:
        figure_days_c_to_lc: output/review/descriptives/figure_hist.svg
        table_csv_summary: output/review/descriptives/summary_days_c_to_long.csv

  ## Figure - hazard ratio plot 

  figure_hazard_ratio:
    run: r:latest analysis/figure_hazard_ratio_plot.R
    needs:
    - development_cox_model_all
    - development_cox_model_vax_c
    - development_cox_model_all_vax_td
    - development_cox_model_infected
    outputs:
      moderately_sensitive:
        figure_hazard_ratio_plot: output/review/model/figure_hr_*.svg

  ## Figure - cumulative probability plot 

  figure_cum_prob_km_all:
    run: r:latest analysis/figure_cum_prob_km.R
    needs:
    - stage1_define_eligible_population_all
    outputs:
      moderately_sensitive:
        cum_prob_plot: output/review/descriptives/figure_cum_*.svg

  ## Suppl_table_1 - frequencies of snomed code for long covid diagnosis 

  suppl_table_1_all:
    run: r:latest analysis/suppl_table_1.R
    needs:
    - stage1_define_eligible_population_all
    outputs:
      moderately_sensitive:
        pie_chart_long_covid_code: output/review/descriptives/suppl_figure_pie.svg
        table_csv_long_covid_code: output/review/descriptives/suppl_table_1.csv
        table_html_long_covid_code: output/review/descriptives/suppl_table_1.html

  ## Suppl_figure_1 - long covid count by region 

  suppl_figure_1_all:
    run: r:latest analysis/suppl_figure_1.R
    needs:
    - stage1_define_eligible_population_all
    outputs:
      moderately_sensitive:
        figure_long_covid_count_region: output/review/descriptives/suppl_figure_1_*.svg
        table_csv_long_covid_count_region: output/review/descriptives/long_covid_count_*.csv
        table_html_long_covid_region: output/review/descriptives/long_covid_count_*.html

  ## Summarise survival data 

  summarise_survival_data_all:
    run: r:latest analysis/stage2_summarise_survival_data.R
    needs:
    - stage1_define_eligible_population_all
    outputs:
      moderately_sensitive:
        summary_survival_data_CSV: output/review/descriptives/summarise_survival_data.csv
        summary_survival_data_HTML: output/review/descriptives/summarise_survival_data.html

  ## Part 4. Modelling 
  ## Development Cox model 
  ## Development Cox Model - all 

  development_cox_model_all:
    run: r:latest analysis/stage3_model_development.R all
    needs:
    - stage1_define_eligible_population_all
    outputs:
      moderately_sensitive:
        ph_test_CSV: output/review/model/PH_test_*_all.csv
        hazard_ratios_CSV: output/review/model/hazard_ratio_estimates_*_all.csv
        hazard_ratios_HTML: output/review/model/hazard_ratio_estimates_*_all.html
        model_selection: output/not_for_review/model/model_selection_all.csv

  ## Development Cox Model - vax_c 

  development_cox_model_vax_c:
    run: r:latest analysis/stage3_model_development.R vax_c
    needs:
    - stage1_define_eligible_population_all
    outputs:
      moderately_sensitive:
        ph_test_CSV: output/review/model/PH_test_*_vax_c.csv
        hazard_ratios_CSV: output/review/model/hazard_ratio_estimates_*_vax_c.csv
        hazard_ratios_HTML: output/review/model/hazard_ratio_estimates_*_vax_c.html
        model_selection: output/not_for_review/model/model_selection_vax_c.csv

  ## Development Cox Model - vaccinated 

  development_cox_model_vaccinated:
    run: r:latest analysis/stage3_model_development.R vaccinated
    needs:
    - stage1_define_eligible_population_vaccinated
    outputs:
      moderately_sensitive:
        ph_test_CSV: output/review/model/PH_test_*_vaccinated.csv
        hazard_ratios_CSV: output/review/model/hazard_ratio_estimates_*_vaccinated.csv
        hazard_ratios_HTML: output/review/model/hazard_ratio_estimates_*_vaccinated.html
        model_selection: output/not_for_review/model/model_selection_vaccinated.csv

  ## Development Cox Model - all_vax_td 

  development_cox_model_all_vax_td:
    run: r:latest analysis/stage3_model_development.R all_vax_td
    needs:
    - stage1_define_eligible_population_all
    outputs:
      moderately_sensitive:
        ph_test_CSV: output/review/model/PH_test_*_all_vax_td.csv
        hazard_ratios_CSV: output/review/model/hazard_ratio_estimates_*_all_vax_td.csv
        hazard_ratios_HTML: output/review/model/hazard_ratio_estimates_*_all_vax_td.html
        model_selection: output/not_for_review/model/model_selection_all_vax_td.csv

  ## Development Cox Model - infected 

  development_cox_model_infected:
    run: r:latest analysis/stage3_model_development.R infected
    needs:
    - stage1_define_eligible_population_infected
    outputs:
      moderately_sensitive:
        ph_test_CSV: output/review/model/PH_test_*_infected.csv
        hazard_ratios_CSV: output/review/model/hazard_ratio_estimates_*_infected.csv
        hazard_ratios_HTML: output/review/model/hazard_ratio_estimates_*_infected.html
        model_selection: output/not_for_review/model/model_selection_infected.csv

  ## Evaluation Cox model 

  evaluation_cox_model_all:
    run: r:latest analysis/stage4_model_evaluation.R all
    needs:
    - stage1_define_eligible_population_all
    outputs:
      moderately_sensitive:
        performance_measure_CSV: output/review/model/performance_measures_*_all.csv
        performance_measure_HTML: output/review/model/performance_measures_*_all.html
        survival_plot: output/review/model/survival_plot_*_all.svg

  evaluation_cox_model_vax_c:
    run: r:latest analysis/stage4_model_evaluation.R vax_c
    needs:
    - stage1_define_eligible_population_all
    outputs:
      moderately_sensitive:
        performance_measure_CSV: output/review/model/performance_measures_*_vax_c.csv
        performance_measure_HTML: output/review/model/performance_measures_*_vax_c.html
        survival_plot: output/review/model/survival_plot_*_vax_c.svg

  evaluation_cox_model_vaccinated:
    run: r:latest analysis/stage4_model_evaluation.R vaccinated
    needs:
    - stage1_define_eligible_population_vaccinated
    outputs:
      moderately_sensitive:
        performance_measure_CSV: output/review/model/performance_measures_*_vaccinated.csv
        performance_measure_HTML: output/review/model/performance_measures_*_vaccinated.html
        survival_plot: output/review/model/survival_plot_*_vaccinated.svg

  evaluation_cox_model_all_vax_td:
    run: r:latest analysis/stage4_model_evaluation.R all_vax_td
    needs:
    - stage1_define_eligible_population_all
    outputs:
      moderately_sensitive:
        performance_measure_CSV: output/review/model/performance_measures_*_all_vax_td.csv
        performance_measure_HTML: output/review/model/performance_measures_*_all_vax_td.html
        survival_plot: output/review/model/survival_plot_*_all_vax_td.svg

  evaluation_cox_model_infected:
    run: r:latest analysis/stage4_model_evaluation.R infected
    needs:
    - stage1_define_eligible_population_infected
    outputs:
      moderately_sensitive:
        performance_measure_CSV: output/review/model/performance_measures_*_infected.csv
        performance_measure_HTML: output/review/model/performance_measures_*_infected.html
        survival_plot: output/review/model/survival_plot_*_infected.svg

  ## Validation Cox Model - all 

  validation_cox_model_all:
    run: r:latest analysis/stage5_model_validation_iecv.R all
    needs:
    - stage1_define_eligible_population_all
    outputs:
      moderately_sensitive:
        val_performance_measure_CSV: output/review/model/val_performance_measures_all.csv
        val_performance_measure_html: output/review/model/val_performance_measures_all.html
        val_cal_plot: output/review/model/val_cal_plot_*_all.svg
        val_re_cal_plot: output/review/model/val_re_cal_plot_*_all.svg

  ## Validation Cox Model - vax_c 

  validation_cox_model_vax_c:
    run: r:latest analysis/stage5_model_validation_iecv.R vax_c
    needs:
    - stage1_define_eligible_population_all
    outputs:
      moderately_sensitive:
        val_performance_measure_CSV: output/review/model/val_performance_measures_vax_c.csv
        val_performance_measure_html: output/review/model/val_performance_measures_vax_c.html
        val_cal_plot: output/review/model/val_cal_plot_*_vax_c.svg
        val_re_cal_plot: output/review/model/val_re_cal_plot_*_vax_c.svg

  ## Validation Cox Model - vaccinated 

  validation_cox_model_vaccinated:
    run: r:latest analysis/stage5_model_validation_iecv.R vaccinated
    needs:
    - stage1_define_eligible_population_vaccinated
    outputs:
      moderately_sensitive:
        val_performance_measure_CSV: output/review/model/val_performance_measures_vaccinated.csv
        val_performance_measure_html: output/review/model/val_performance_measures_vaccinated.html
        val_cal_plot: output/review/model/val_cal_plot_*_vaccinated.svg
        val_re_cal_plot: output/review/model/val_re_cal_plot_*_vaccinated.svg

  ## Validation Cox Model - all_vax_td 

  validation_cox_model_all_vax_td:
    run: r:latest analysis/stage5_model_validation_iecv.R all_vax_td
    needs:
    - stage1_define_eligible_population_all
    outputs:
      moderately_sensitive:
        val_performance_measure_CSV: output/review/model/val_performance_measures_all_vax_td.csv
        val_performance_measure_html: output/review/model/val_performance_measures_all_vax_td.html
        val_cal_plot: output/review/model/val_cal_plot_*_all_vax_td.svg
        val_re_cal_plot: output/review/model/val_re_cal_plot_*_all_vax_td.svg

  ## Validation Cox Model - infected 

  validation_cox_model_infected:
    run: r:latest analysis/stage5_model_validation_iecv.R infected
    needs:
    - stage1_define_eligible_population_infected
    outputs:
      moderately_sensitive:
        val_performance_measure_CSV: output/review/model/val_performance_measures_infected.csv
        val_performance_measure_html: output/review/model/val_performance_measures_infected.html
        val_cal_plot: output/review/model/val_cal_plot_*_infected.svg
        val_re_cal_plot: output/review/model/val_re_cal_plot_*_infected.svg

State

State is inferred from the related Jobs.

Status: Failed

Timings

Timings set to UTC timezone.

  • Created:
  • Started:
  • Finished:
  • Runtime: 00:25:14

Config

  • Backend:
    TPP
  • Workspace:
    long-covid-risk-factors
  • Branch:
    main
  • Creator:
    yinghuiwei
  • Force run dependencies:
    False
  • Git Commit Hash:
    e79e647
  • Requested actions:
    • stage0_data_cleaning_all
    • stage1_define_eligible_population_all
    • table_1_all
    • table_2_all
    • table_3_all
    • figure_1_all
    • figure_2_all
    • figure_cum_prob_km_all
    • suppl_table_1_all
    • suppl_figure_1_all
    • summarise_survival_data_all
    • development_cox_model_all
    • development_cox_model_vax_c
    • development_cox_model_all_vax_td
    • evaluation_cox_model_all
    • evaluation_cox_model_vax_c
    • evaluation_cox_model_all_vax_td
    • validation_cox_model_all
    • validation_cox_model_vax_c
    • validation_cox_model_all_vax_td