Job request: 8992

View Repo View project.yaml

This page shows the technical details of what happened when authorised researcher Yinghui Wei requested one or more actions to be run against real patient data in the long-covid-risk-factors project, within a secure environment.

By cross-referencing the indicated Requested Actions with the Pipeline section below, you can infer what security level various outputs were written to. Outputs marked as highly_sensitive can never be viewed directly by a researcher; they can only request that code runs against them. Outputs marked as moderately_sensitive can be viewed by an approved researcher by logging into a highly secure environment. Only outputs marked as moderately_sensitive can be requested for release to the public, via a controlled output review service.

Jobs

ID Status Action
yi4qs7ino2sauz6v succeeded development_cox_model_all
e7m6cx7wj4nibcuz succeeded development_cox_model_all_vax_td
gsziad4ydceomn7p failed development_cox_model_infected
4fji6wvwdg2uxsbl succeeded development_cox_model_vaccinated
phurtge35zvlh6j4 succeeded evaluation_cox_model_all
hrqu7j7znogxw4q5 succeeded evaluation_cox_model_all_vax_td
hera7xzlk6zxfymg succeeded evaluation_cox_model_infected
zgt7spktevbcvspy succeeded evaluation_cox_model_vaccinated
bso7zk36yjbgdjqq failed figure_hazard_ratio
am33wq5qfy4ma6wb succeeded generate_study_population_all
wmeuvnhri4mn5c5p succeeded generate_study_population_infected
wocbvtj3eysrsedx succeeded generate_study_population_vaccinated
5ehcslib64e6m73i succeeded stage0_data_cleaning_all
65nzjpjojzujkvtw succeeded stage0_data_cleaning_infected
ub6qu2emnfn7qgf6 succeeded stage1_define_eligible_population_infected
ogs2die4nubi5h2g succeeded table_1_infected
u5se2et63m3rwrj7 succeeded validation_cox_model_all
ttes7p7xgmks22pq succeeded validation_cox_model_all_vax_td
3ftav7xcr5ahi45o succeeded validation_cox_model_infected
qa2ix3v6tw5nhwoq succeeded validation_cox_model_vaccinated
57r363yznbbs5uvy succeeded stage0_data_cleaning_vaccinated
oti3n2bj6tz2wb2s succeeded stage1_define_eligible_population_all
soc354j7nesczggr succeeded stage1_define_eligible_population_vaccinated
qg33d535pnuatgis succeeded table_1_all
w3epvzkwcmevhtjm succeeded table_1_vaccinated
xmsayihgfpnc7ad2 succeeded table_2_all
s4kc7buuxyrrbifr succeeded table_2_vaccinated
5er3p7ysapvelpid succeeded table_2_infected
dnoeqein7chemxgu succeeded table_3_all
dotdswitgnvj54vr succeeded figure_1_all
sbtsyysazk5cxanr succeeded figure_2_all
wibchbbpbrt4v3ri failed development_cox_model_all_vax_c
fm5xprc3rf2fm7g2 succeeded figure_cum_prob_km_all
g3kfo2o7boqs46rg succeeded suppl_table_1_all
4midsvghuhwiem4n succeeded suppl_figure_1_all
2cbopgzzsyznnwis succeeded summarise_survival_data_all
mosoztw3etbbp3kq succeeded evaluation_cox_model_all_vax_c
nggkef2jt5ddccaq succeeded validation_cox_model_all_vax_c

Pipeline

Show Hide project.yaml
version: '3.0'

expectations:

  population_size: 15000

actions:

  ## # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # 
  ## DO NOT EDIT project.yaml DIRECTLY 
  ## This file is created by create_project_actions.R 
  ## Edit and run create_project_actions.R to update the project.yaml 
  ## # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # 
  ##  
  ## Part 1. Generate Study Population 
  ##  
  ## Generate dummy data for all eligible population 

  generate_study_population_all:
    run: cohortextractor:latest generate_cohort --study-definition study_definition_all
      --output-format feather
    outputs:
      highly_sensitive:
        cohort: output/input_all.feather

  ## Generate dummy data for study population - vaccinated 

  generate_study_population_vaccinated:
    run: cohortextractor:latest generate_cohort --study-definition study_definition_vaccinated
      --output-format feather
    outputs:
      highly_sensitive:
        cohort: output/input_vaccinated.feather

  ## Generate dummy data for study population - infected 

  generate_study_population_infected:
    run: cohortextractor:latest generate_cohort --study-definition study_definition_infected
      --output-format feather
    outputs:
      highly_sensitive:
        cohort: output/input_infected.feather

  ## Part 2. Create Analysis Data Sets 
  ## Data Cleaning 
  ## Stage 0 Data Cleaning - all 

  stage0_data_cleaning_all:
    run: r:latest analysis/stage0_data_cleaning.R all
    needs:
    - generate_study_population_all
    outputs:
      highly_sensitive:
        cohort: output/input_stage0_all.rds
      moderately_sensitive:
        variable_check_table_CSV: output/not_for_review/descriptives/table_0_all.csv
        variable_check_table_HTML: output/not_for_review/descriptives/table_0_all.html
        histogram_numerical_variable: output/not_for_review/descriptives/histogram_*_all.svg

  ## Stage 0 Data Cleaning - vaccinated 

  stage0_data_cleaning_vaccinated:
    run: r:latest analysis/stage0_data_cleaning.R vaccinated
    needs:
    - generate_study_population_vaccinated
    outputs:
      highly_sensitive:
        cohort: output/input_stage0_vaccinated.rds
      moderately_sensitive:
        variable_check_table_CSV: output/not_for_review/descriptives/table_0_vaccinated.csv
        variable_check_table_HTML: output/not_for_review/descriptives/table_0_vaccinated.html
        histogram_numerical_variable: output/not_for_review/descriptives/histogram_*_vaccinated.svg

  ## Stage 0 Data Cleaning - infected 

  stage0_data_cleaning_infected:
    run: r:latest analysis/stage0_data_cleaning.R infected
    needs:
    - generate_study_population_infected
    outputs:
      highly_sensitive:
        cohort: output/input_stage0_infected.rds
      moderately_sensitive:
        variable_check_table_CSV: output/not_for_review/descriptives/table_0_infected.csv
        variable_check_table_HTML: output/not_for_review/descriptives/table_0_infected.html
        histogram_numerical_variable: output/not_for_review/descriptives/histogram_*_infected.svg

  ## Define eligible population 
  ## Stage 1 define eligible population - all 

  stage1_define_eligible_population_all:
    run: r:latest analysis/stage1_define_eligible_population.R all
    needs:
    - stage0_data_cleaning_all
    outputs:
      highly_sensitive:
        cohort: output/input_stage1_all.rds
      moderately_sensitive:
        flow_chart_csv: output/flow_chart_all.csv
        flow_chart_html: output/flow_chart_all.html

  ## Stage 1 define eligible population - vaccinated 

  stage1_define_eligible_population_vaccinated:
    run: r:latest analysis/stage1_define_eligible_population.R vaccinated
    needs:
    - stage0_data_cleaning_vaccinated
    outputs:
      highly_sensitive:
        cohort: output/input_stage1_vaccinated.rds
      moderately_sensitive:
        flow_chart_csv: output/flow_chart_vaccinated.csv
        flow_chart_html: output/flow_chart_vaccinated.html

  ## Stage 1 define eligible population - infected 

  stage1_define_eligible_population_infected:
    run: r:latest analysis/stage1_define_eligible_population.R infected
    needs:
    - stage0_data_cleaning_infected
    outputs:
      highly_sensitive:
        cohort: output/input_stage1_infected.rds
      moderately_sensitive:
        flow_chart_csv: output/flow_chart_infected.csv
        flow_chart_html: output/flow_chart_infected.html

  ## Part 3. Figures and tables 
  ## Table 1. Patient Characteristics - all 

  table_1_all:
    run: r:latest analysis/table_1.R all
    needs:
    - stage1_define_eligible_population_all
    outputs:
      moderately_sensitive:
        descriptive_table_CSV: output/review/descriptives/table_1_all.csv
        descriptive_table_HTML: output/review/descriptives/table_1_all.html

  ## Table 1. Patient Characteristics - vaccinated 

  table_1_vaccinated:
    run: r:latest analysis/table_1.R vaccinated
    needs:
    - stage1_define_eligible_population_vaccinated
    outputs:
      moderately_sensitive:
        descriptive_table_CSV: output/review/descriptives/table_1_vaccinated.csv
        descriptive_table_HTML: output/review/descriptives/table_1_vaccinated.html

  ## Table 1. Patient Characteristics - infected 

  table_1_infected:
    run: r:latest analysis/table_1.R infected
    needs:
    - stage1_define_eligible_population_infected
    outputs:
      moderately_sensitive:
        descriptive_table_CSV: output/review/descriptives/table_1_infected.csv
        descriptive_table_HTML: output/review/descriptives/table_1_infected.html

  ## Define eligible population 
  ## Table 2. Event count and incidence rate - all 

  table_2_all:
    run: r:latest analysis/table_2.R all
    needs:
    - stage1_define_eligible_population_all
    outputs:
      moderately_sensitive:
        incidence_rate_table_CSV: output/review/descriptives/table_2_all.csv
        incidence_rate_talbe_HTML: output/review/descriptives/table_2_all.html

  ## Table 2. Event count and incidence rate - vaccinated 

  table_2_vaccinated:
    run: r:latest analysis/table_2.R vaccinated
    needs:
    - stage1_define_eligible_population_vaccinated
    outputs:
      moderately_sensitive:
        incidence_rate_table_CSV: output/review/descriptives/table_2_vaccinated.csv
        incidence_rate_talbe_HTML: output/review/descriptives/table_2_vaccinated.html

  ## Table 2. Event count and incidence rate - infected 

  table_2_infected:
    run: r:latest analysis/table_2.R infected
    needs:
    - stage1_define_eligible_population_infected
    outputs:
      moderately_sensitive:
        incidence_rate_table_CSV: output/review/descriptives/table_2_infected.csv
        incidence_rate_talbe_HTML: output/review/descriptives/table_2_infected.html

  ## table_3 - sequence count 

  table_3_all:
    run: r:latest analysis/table_3.R
    needs:
    - stage1_define_eligible_population_all
    outputs:
      moderately_sensitive:
        sequence_count_table_CSV: output/review/descriptives/table_3.csv
        sequence_count_table_HTML: output/review/descriptives/table_3.html

  ## Figure_1 - long covid count 

  figure_1_all:
    run: r:latest analysis/figure_1.R
    needs:
    - stage1_define_eligible_population_all
    outputs:
      moderately_sensitive:
        figure_long_covid_count_all: output/figure_1_*.svg
        table_csv_long_covid_count_all: output/review/descriptives/long_covid_count_*_all.csv
        table_html_long_covid_count_all: output/review/descriptives/long_covid_count_*_all.html

  ## Figure_2 - days from covid to long covid 

  figure_2_all:
    run: r:latest analysis/figure_2.R
    needs:
    - stage1_define_eligible_population_all
    outputs:
      moderately_sensitive:
        figure_days_c_to_lc: output/review/descriptives/figure_hist.svg
        table_csv_summary: output/review/descriptives/summary_days_c_to_long.csv

  ## Figure - hazard ratio plot 

  figure_hazard_ratio:
    run: r:latest analysis/figure_hazard_ratio_plot.R
    needs:
    - development_cox_model_all
    - development_cox_model_all_vax_c
    - development_cox_model_all_vax_td
    - development_cox_model_infected
    outputs:
      moderately_sensitive:
        figure_hazard_ratio_plot: output/review/model/figure_hr_*.svg

  ## Figure - cumulative probability plot 

  figure_cum_prob_km_all:
    run: r:latest analysis/figure_cum_prob_km.R
    needs:
    - stage1_define_eligible_population_all
    outputs:
      moderately_sensitive:
        cum_prob_plot: output/review/descriptives/figure_cum_*.svg

  ## Suppl_table_1 - frequencies of snomed code for long covid diagnosis 

  suppl_table_1_all:
    run: r:latest analysis/suppl_table_1.R
    needs:
    - stage1_define_eligible_population_all
    outputs:
      moderately_sensitive:
        pie_chart_long_covid_code: output/review/descriptives/suppl_figure_pie.svg
        table_csv_long_covid_code: output/review/descriptives/suppl_table_1.csv
        table_html_long_covid_code: output/review/descriptives/suppl_table_1.html

  ## Suppl_figure_1 - long covid count by region 

  suppl_figure_1_all:
    run: r:latest analysis/suppl_figure_1.R
    needs:
    - stage1_define_eligible_population_all
    outputs:
      moderately_sensitive:
        figure_long_covid_count_region: output/review/descriptives/suppl_figure_1_*.svg
        table_csv_long_covid_count_region: output/review/descriptives/long_covid_count_*.csv
        table_html_long_covid_region: output/review/descriptives/long_covid_count_*.html

  ## Summarise survival data 

  summarise_survival_data_all:
    run: r:latest analysis/stage2_summarise_survival_data.R
    needs:
    - stage1_define_eligible_population_all
    outputs:
      moderately_sensitive:
        summary_survival_data_CSV: output/review/descriptives/summarise_survival_data.csv
        summary_survival_data_HTML: output/review/descriptives/summarise_survival_data.html

  ## Part 4. Modelling 
  ## Development Cox model 
  ## Development Cox Model - all 

  development_cox_model_all:
    run: r:latest analysis/stage3_model_development.R all
    needs:
    - stage1_define_eligible_population_all
    outputs:
      moderately_sensitive:
        ph_test_CSV: output/review/model/PH_test_*_all.csv
        hazard_ratios_CSV: output/review/model/hazard_ratio_estimates_*_all.csv
        hazard_ratios_HTML: output/review/model/hazard_ratio_estimates_*_all.html
        model_selection: output/not_for_review/model/model_selection_all.csv

  ## Development Cox Model - all_vax_c 

  development_cox_model_all_vax_c:
    run: r:latest analysis/stage3_model_development.R all_vax_c
    needs:
    - stage1_define_eligible_population_all
    outputs:
      moderately_sensitive:
        ph_test_CSV: output/review/model/PH_test_*_all_vax_c.csv
        hazard_ratios_CSV: output/review/model/hazard_ratio_estimates_*_all_vax_c.csv
        hazard_ratios_HTML: output/review/model/hazard_ratio_estimates_*_all_vax_c.html
        model_selection: output/not_for_review/model/model_selection_all_vax_c.csv

  ## Development Cox Model - vaccinated 

  development_cox_model_vaccinated:
    run: r:latest analysis/stage3_model_development.R vaccinated
    needs:
    - stage1_define_eligible_population_vaccinated
    outputs:
      moderately_sensitive:
        ph_test_CSV: output/review/model/PH_test_*_vaccinated.csv
        hazard_ratios_CSV: output/review/model/hazard_ratio_estimates_*_vaccinated.csv
        hazard_ratios_HTML: output/review/model/hazard_ratio_estimates_*_vaccinated.html
        model_selection: output/not_for_review/model/model_selection_vaccinated.csv

  ## Development Cox Model - all_vax_td 

  development_cox_model_all_vax_td:
    run: r:latest analysis/stage3_model_development.R all_vax_td
    needs:
    - stage1_define_eligible_population_all
    outputs:
      moderately_sensitive:
        ph_test_CSV: output/review/model/PH_test_*_all_vax_td.csv
        hazard_ratios_CSV: output/review/model/hazard_ratio_estimates_*_all_vax_td.csv
        hazard_ratios_HTML: output/review/model/hazard_ratio_estimates_*_all_vax_td.html
        model_selection: output/not_for_review/model/model_selection_all_vax_td.csv

  ## Development Cox Model - infected 

  development_cox_model_infected:
    run: r:latest analysis/stage3_model_development.R infected
    needs:
    - stage1_define_eligible_population_infected
    outputs:
      moderately_sensitive:
        ph_test_CSV: output/review/model/PH_test_*_infected.csv
        hazard_ratios_CSV: output/review/model/hazard_ratio_estimates_*_infected.csv
        hazard_ratios_HTML: output/review/model/hazard_ratio_estimates_*_infected.html
        model_selection: output/not_for_review/model/model_selection_infected.csv

  ## Evaluation Cox model 

  evaluation_cox_model_all:
    run: r:latest analysis/stage4_model_evaluation.R all
    needs:
    - stage1_define_eligible_population_all
    outputs:
      moderately_sensitive:
        performance_measure_CSV: output/review/model/performance_measures_*_all.csv
        performance_measure_HTML: output/review/model/performance_measures_*_all.html
        survival_plot: output/review/model/survival_plot_*_all.svg

  evaluation_cox_model_all_vax_c:
    run: r:latest analysis/stage4_model_evaluation.R all_vax_c
    needs:
    - stage1_define_eligible_population_all
    outputs:
      moderately_sensitive:
        performance_measure_CSV: output/review/model/performance_measures_*_all_vax_c.csv
        performance_measure_HTML: output/review/model/performance_measures_*_all_vax_c.html
        survival_plot: output/review/model/survival_plot_*_all_vax_c.svg

  evaluation_cox_model_vaccinated:
    run: r:latest analysis/stage4_model_evaluation.R vaccinated
    needs:
    - stage1_define_eligible_population_vaccinated
    outputs:
      moderately_sensitive:
        performance_measure_CSV: output/review/model/performance_measures_*_vaccinated.csv
        performance_measure_HTML: output/review/model/performance_measures_*_vaccinated.html
        survival_plot: output/review/model/survival_plot_*_vaccinated.svg

  evaluation_cox_model_all_vax_td:
    run: r:latest analysis/stage4_model_evaluation.R all_vax_td
    needs:
    - stage1_define_eligible_population_all
    outputs:
      moderately_sensitive:
        performance_measure_CSV: output/review/model/performance_measures_*_all_vax_td.csv
        performance_measure_HTML: output/review/model/performance_measures_*_all_vax_td.html
        survival_plot: output/review/model/survival_plot_*_all_vax_td.svg

  evaluation_cox_model_infected:
    run: r:latest analysis/stage4_model_evaluation.R infected
    needs:
    - stage1_define_eligible_population_infected
    outputs:
      moderately_sensitive:
        performance_measure_CSV: output/review/model/performance_measures_*_infected.csv
        performance_measure_HTML: output/review/model/performance_measures_*_infected.html
        survival_plot: output/review/model/survival_plot_*_infected.svg

  ## Validation Cox Model - all 

  validation_cox_model_all:
    run: r:latest analysis/stage5_model_validation_iecv.R all
    needs:
    - stage1_define_eligible_population_all
    outputs:
      moderately_sensitive:
        val_performance_measure_CSV: output/review/model/val_performance_measures_all.csv
        val_performance_measure_html: output/review/model/val_performance_measures_all.html
        val_cal_plot: output/review/model/val_cal_plot_*_all.svg
        val_re_cal_plot: output/review/model/val_re_cal_plot_*_all.svg

  ## Validation Cox Model - all_vax_c 

  validation_cox_model_all_vax_c:
    run: r:latest analysis/stage5_model_validation_iecv.R all_vax_c
    needs:
    - stage1_define_eligible_population_all
    outputs:
      moderately_sensitive:
        val_performance_measure_CSV: output/review/model/val_performance_measures_all_vax_c.csv
        val_performance_measure_html: output/review/model/val_performance_measures_all_vax_c.html
        val_cal_plot: output/review/model/val_cal_plot_*_all_vax_c.svg
        val_re_cal_plot: output/review/model/val_re_cal_plot_*_all_vax_c.svg

  ## Validation Cox Model - vaccinated 

  validation_cox_model_vaccinated:
    run: r:latest analysis/stage5_model_validation_iecv.R vaccinated
    needs:
    - stage1_define_eligible_population_vaccinated
    outputs:
      moderately_sensitive:
        val_performance_measure_CSV: output/review/model/val_performance_measures_vaccinated.csv
        val_performance_measure_html: output/review/model/val_performance_measures_vaccinated.html
        val_cal_plot: output/review/model/val_cal_plot_*_vaccinated.svg
        val_re_cal_plot: output/review/model/val_re_cal_plot_*_vaccinated.svg

  ## Validation Cox Model - all_vax_td 

  validation_cox_model_all_vax_td:
    run: r:latest analysis/stage5_model_validation_iecv.R all_vax_td
    needs:
    - stage1_define_eligible_population_all
    outputs:
      moderately_sensitive:
        val_performance_measure_CSV: output/review/model/val_performance_measures_all_vax_td.csv
        val_performance_measure_html: output/review/model/val_performance_measures_all_vax_td.html
        val_cal_plot: output/review/model/val_cal_plot_*_all_vax_td.svg
        val_re_cal_plot: output/review/model/val_re_cal_plot_*_all_vax_td.svg

  ## Validation Cox Model - infected 

  validation_cox_model_infected:
    run: r:latest analysis/stage5_model_validation_iecv.R infected
    needs:
    - stage1_define_eligible_population_infected
    outputs:
      moderately_sensitive:
        val_performance_measure_CSV: output/review/model/val_performance_measures_infected.csv
        val_performance_measure_html: output/review/model/val_performance_measures_infected.html
        val_cal_plot: output/review/model/val_cal_plot_*_infected.svg
        val_re_cal_plot: output/review/model/val_re_cal_plot_*_infected.svg

State

State is inferred from the related Jobs.

Status: Failed

Timings

Timings set to UTC timezone.

  • Created:
  • Started:
  • Finished:
  • Runtime: 81:26:00

Config