Job request: 9234

View Repo View project.yaml

This page shows the technical details of what happened when authorised researcher Yinghui Wei requested one or more actions to be run against real patient data in the long-covid-risk-factors project, within a secure environment.

By cross-referencing the indicated Requested Actions with the Pipeline section below, you can infer what security level various outputs were written to. Outputs marked as highly_sensitive can never be viewed directly by a researcher; they can only request that code runs against them. Outputs marked as moderately_sensitive can be viewed by an approved researcher by logging into a highly secure environment. Only outputs marked as moderately_sensitive can be requested for release to the public, via a controlled output review service.

Jobs

ID Status Action
lvitz7lnqbc54tm3 succeeded development_cox_model_all
do2dktkmdxz54lwg succeeded development_cox_model_all_vax_c
xz6iosbtcz4md6hj succeeded development_cox_model_all_vax_td
f7szced7chug6ah3 succeeded development_cox_model_infected
f7q7wjfmvkq6uawi succeeded development_cox_model_vaccinated

Pipeline

Show Hide project.yaml
version: '3.0'

expectations:

  population_size: 15000

actions:

  ## # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # 
  ## DO NOT EDIT project.yaml DIRECTLY 
  ## This file is created by create_project_actions.R 
  ## Edit and run create_project_actions.R to update the project.yaml 
  ## # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # 
  ##  
  ## Part 1. Generate Study Population 
  ##  
  ## Generate dummy data for all eligible population 

  generate_study_population_all:
    run: cohortextractor:latest generate_cohort --study-definition study_definition_all
      --output-format feather
    outputs:
      highly_sensitive:
        cohort: output/input_all.feather

  ## Generate dummy data for study population - vaccinated 

  generate_study_population_vaccinated:
    run: cohortextractor:latest generate_cohort --study-definition study_definition_vaccinated
      --output-format feather
    outputs:
      highly_sensitive:
        cohort: output/input_vaccinated.feather

  ## Generate dummy data for study population - infected 

  generate_study_population_infected:
    run: cohortextractor:latest generate_cohort --study-definition study_definition_infected
      --output-format feather
    outputs:
      highly_sensitive:
        cohort: output/input_infected.feather

  ## Part 2. Create Analysis Data Sets 
  ## Data Cleaning 
  ## Stage 0 Data Cleaning - all 

  stage0_data_cleaning_all:
    run: r:latest analysis/stage0_data_cleaning.R all
    needs:
    - generate_study_population_all
    outputs:
      highly_sensitive:
        cohort: output/input_stage0_all.rds
      moderately_sensitive:
        variable_check_table_CSV: output/not_for_review/descriptives/table_0_all.csv
        variable_check_table_HTML: output/not_for_review/descriptives/table_0_all.html
        histogram_numerical_variable: output/not_for_review/descriptives/histogram_*_all.svg

  ## Stage 0 Data Cleaning - vaccinated 

  stage0_data_cleaning_vaccinated:
    run: r:latest analysis/stage0_data_cleaning.R vaccinated
    needs:
    - generate_study_population_vaccinated
    outputs:
      highly_sensitive:
        cohort: output/input_stage0_vaccinated.rds
      moderately_sensitive:
        variable_check_table_CSV: output/not_for_review/descriptives/table_0_vaccinated.csv
        variable_check_table_HTML: output/not_for_review/descriptives/table_0_vaccinated.html
        histogram_numerical_variable: output/not_for_review/descriptives/histogram_*_vaccinated.svg

  ## Stage 0 Data Cleaning - infected 

  stage0_data_cleaning_infected:
    run: r:latest analysis/stage0_data_cleaning.R infected
    needs:
    - generate_study_population_infected
    outputs:
      highly_sensitive:
        cohort: output/input_stage0_infected.rds
      moderately_sensitive:
        variable_check_table_CSV: output/not_for_review/descriptives/table_0_infected.csv
        variable_check_table_HTML: output/not_for_review/descriptives/table_0_infected.html
        histogram_numerical_variable: output/not_for_review/descriptives/histogram_*_infected.svg

  ## Define eligible population 
  ## Stage 1 define eligible population - all 

  stage1_define_eligible_population_all:
    run: r:latest analysis/stage1_define_eligible_population.R all
    needs:
    - stage0_data_cleaning_all
    outputs:
      highly_sensitive:
        cohort: output/input_stage1_all.rds
      moderately_sensitive:
        flow_chart_csv: output/flow_chart_all.csv
        flow_chart_html: output/flow_chart_all.html

  ## Stage 1 define eligible population - vaccinated 

  stage1_define_eligible_population_vaccinated:
    run: r:latest analysis/stage1_define_eligible_population.R vaccinated
    needs:
    - stage0_data_cleaning_vaccinated
    outputs:
      highly_sensitive:
        cohort: output/input_stage1_vaccinated.rds
      moderately_sensitive:
        flow_chart_csv: output/flow_chart_vaccinated.csv
        flow_chart_html: output/flow_chart_vaccinated.html

  ## Stage 1 define eligible population - infected 

  stage1_define_eligible_population_infected:
    run: r:latest analysis/stage1_define_eligible_population.R infected
    needs:
    - stage0_data_cleaning_infected
    outputs:
      highly_sensitive:
        cohort: output/input_stage1_infected.rds
      moderately_sensitive:
        flow_chart_csv: output/flow_chart_infected.csv
        flow_chart_html: output/flow_chart_infected.html

  ## Part 3. Figures and tables 
  ## Table 1. Patient Characteristics - all 

  table_1_all:
    run: r:latest analysis/table_1.R all
    needs:
    - stage1_define_eligible_population_all
    outputs:
      moderately_sensitive:
        descriptive_table_CSV: output/review/descriptives/table_1_all.csv
        descriptive_table_HTML: output/review/descriptives/table_1_all.html

  ## Table 1. Patient Characteristics - vaccinated 

  table_1_vaccinated:
    run: r:latest analysis/table_1.R vaccinated
    needs:
    - stage1_define_eligible_population_vaccinated
    outputs:
      moderately_sensitive:
        descriptive_table_CSV: output/review/descriptives/table_1_vaccinated.csv
        descriptive_table_HTML: output/review/descriptives/table_1_vaccinated.html

  ## Table 1. Patient Characteristics - infected 

  table_1_infected:
    run: r:latest analysis/table_1.R infected
    needs:
    - stage1_define_eligible_population_infected
    outputs:
      moderately_sensitive:
        descriptive_table_CSV: output/review/descriptives/table_1_infected.csv
        descriptive_table_HTML: output/review/descriptives/table_1_infected.html

  ## Define eligible population 
  ## Table 2. Event count and incidence rate - all 

  table_2_all:
    run: r:latest analysis/table_2.R all
    needs:
    - stage1_define_eligible_population_all
    outputs:
      moderately_sensitive:
        incidence_rate_table_CSV: output/review/descriptives/table_2_all.csv
        incidence_rate_talbe_HTML: output/review/descriptives/table_2_all.html

  ## Table 2. Event count and incidence rate - vaccinated 

  table_2_vaccinated:
    run: r:latest analysis/table_2.R vaccinated
    needs:
    - stage1_define_eligible_population_vaccinated
    outputs:
      moderately_sensitive:
        incidence_rate_table_CSV: output/review/descriptives/table_2_vaccinated.csv
        incidence_rate_talbe_HTML: output/review/descriptives/table_2_vaccinated.html

  ## Table 2. Event count and incidence rate - infected 

  table_2_infected:
    run: r:latest analysis/table_2.R infected
    needs:
    - stage1_define_eligible_population_infected
    outputs:
      moderately_sensitive:
        incidence_rate_table_CSV: output/review/descriptives/table_2_infected.csv
        incidence_rate_talbe_HTML: output/review/descriptives/table_2_infected.html

  ## table_3 - sequence count 

  table_3_all:
    run: r:latest analysis/table_3.R
    needs:
    - stage1_define_eligible_population_all
    outputs:
      moderately_sensitive:
        sequence_count_table_CSV: output/review/descriptives/table_3.csv
        sequence_count_table_HTML: output/review/descriptives/table_3.html

  ## Figure_1 - long covid count 

  figure_1_all:
    run: r:latest analysis/figure_1.R
    needs:
    - stage1_define_eligible_population_all
    outputs:
      moderately_sensitive:
        figure_long_covid_count_all: output/figure_1_*.svg
        table_csv_long_covid_count_all: output/review/descriptives/long_covid_count_*_all.csv
        table_html_long_covid_count_all: output/review/descriptives/long_covid_count_*_all.html

  ## Figure_hist - Histogram of days from covid to long covid 

  figure_hist_all:
    run: r:latest analysis/figure_hist.R
    needs:
    - stage1_define_eligible_population_all
    outputs:
      moderately_sensitive:
        figure_days_c_to_lc: output/review/descriptives/figure_hist.svg
        table_csv_summary: output/review/descriptives/summary_days_c_to_long.csv
        table_bin_count: output/review/descriptives/hist_*

  ## Figure - hazard ratio plot 

  figure_hazard_ratio:
    run: r:latest analysis/figure_hazard_ratio_plot.R
    needs:
    - development_cox_model_all
    - development_cox_model_all_vax_c
    - development_cox_model_all_vax_td
    - development_cox_model_infected
    outputs:
      moderately_sensitive:
        figure_hazard_ratio_plot: output/review/model/figure_hr_*.svg

  ## Figure - cumulative probability plot 

  figure_cum_prob_km_all:
    run: r:latest analysis/figure_cum_prob_km.R
    needs:
    - stage1_define_eligible_population_all
    outputs:
      moderately_sensitive:
        cum_prob_plot: output/review/descriptives/figure_cum_*.svg

  ## Suppl_table_1 - frequencies of snomed code for long covid diagnosis 

  suppl_table_1_all:
    run: r:latest analysis/suppl_table_1.R
    needs:
    - stage1_define_eligible_population_all
    outputs:
      moderately_sensitive:
        pie_chart_long_covid_code: output/review/descriptives/suppl_figure_pie.svg
        table_csv_long_covid_code: output/review/descriptives/suppl_table_1.csv
        table_html_long_covid_code: output/review/descriptives/suppl_table_1.html

  ## Suppl_figure_1 - long covid count by region 

  suppl_figure_1_all:
    run: r:latest analysis/suppl_figure_1.R
    needs:
    - stage1_define_eligible_population_all
    outputs:
      moderately_sensitive:
        figure_long_covid_count_region: output/review/descriptives/suppl_figure_1_*.svg
        table_csv_long_covid_count_region: output/review/descriptives/long_covid_count_*.csv
        table_html_long_covid_region: output/review/descriptives/long_covid_count_*.html

  ## Summarise survival data 

  summarise_survival_data_all:
    run: r:latest analysis/stage2_summarise_survival_data.R
    needs:
    - stage1_define_eligible_population_all
    outputs:
      moderately_sensitive:
        summary_survival_data_CSV: output/review/descriptives/summarise_survival_data.csv
        summary_survival_data_HTML: output/review/descriptives/summarise_survival_data.html

  ## Part 4. Modelling 
  ## Development Cox model 
  ## Development Cox Model - all 

  development_cox_model_all:
    run: r:latest analysis/stage3_model_development.R all
    needs:
    - stage1_define_eligible_population_all
    outputs:
      moderately_sensitive:
        supporting_document_CSV: output/review/model/analysis_data_summary_all.csv
        supporting_document_html: output/review/model/analysis_data_summary_all.html
        hazard_ratios_CSV: output/review/model/hazard_ratio_estimates_*_all.csv
        hazard_ratios_HTML: output/review/model/hazard_ratio_estimates_*_all.html
        model_selection: output/not_for_review/model/model_selection_all.csv

  ## Development Cox Model - all_vax_c 

  development_cox_model_all_vax_c:
    run: r:latest analysis/stage3_model_development.R all_vax_c
    needs:
    - stage1_define_eligible_population_all
    outputs:
      moderately_sensitive:
        supporting_document_CSV: output/review/model/analysis_data_summary_all_vax_c.csv
        supporting_document_html: output/review/model/analysis_data_summary_all_vax_c.html
        hazard_ratios_CSV: output/review/model/hazard_ratio_estimates_*_all_vax_c.csv
        hazard_ratios_HTML: output/review/model/hazard_ratio_estimates_*_all_vax_c.html
        model_selection: output/not_for_review/model/model_selection_all_vax_c.csv

  ## Development Cox Model - vaccinated 

  development_cox_model_vaccinated:
    run: r:latest analysis/stage3_model_development.R vaccinated
    needs:
    - stage1_define_eligible_population_vaccinated
    outputs:
      moderately_sensitive:
        supporting_document_CSV: output/review/model/analysis_data_summary_vaccinated.csv
        supporting_document_html: output/review/model/analysis_data_summary_vaccinated.html
        hazard_ratios_CSV: output/review/model/hazard_ratio_estimates_*_vaccinated.csv
        hazard_ratios_HTML: output/review/model/hazard_ratio_estimates_*_vaccinated.html
        model_selection: output/not_for_review/model/model_selection_vaccinated.csv

  ## Development Cox Model - all_vax_td 

  development_cox_model_all_vax_td:
    run: r:latest analysis/stage3_model_development.R all_vax_td
    needs:
    - stage1_define_eligible_population_all
    outputs:
      moderately_sensitive:
        supporting_document_CSV: output/review/model/analysis_data_summary_all_vax_td.csv
        supporting_document_html: output/review/model/analysis_data_summary_all_vax_td.html
        hazard_ratios_CSV: output/review/model/hazard_ratio_estimates_*_all_vax_td.csv
        hazard_ratios_HTML: output/review/model/hazard_ratio_estimates_*_all_vax_td.html
        model_selection: output/not_for_review/model/model_selection_all_vax_td.csv

  ## Development Cox Model - infected 

  development_cox_model_infected:
    run: r:latest analysis/stage3_model_development.R infected
    needs:
    - stage1_define_eligible_population_infected
    outputs:
      moderately_sensitive:
        supporting_document_CSV: output/review/model/analysis_data_summary_infected.csv
        supporting_document_html: output/review/model/analysis_data_summary_infected.html
        hazard_ratios_CSV: output/review/model/hazard_ratio_estimates_*_infected.csv
        hazard_ratios_HTML: output/review/model/hazard_ratio_estimates_*_infected.html
        model_selection: output/not_for_review/model/model_selection_infected.csv

  ## Evaluation Cox model 

  evaluation_cox_model_all:
    run: r:latest analysis/stage4_model_evaluation.R all
    needs:
    - stage1_define_eligible_population_all
    outputs:
      moderately_sensitive:
        performance_measure_CSV: output/review/model/performance_measures_*_all.csv
        performance_measure_HTML: output/review/model/performance_measures_*_all.html
        survival_plot: output/review/model/survival_plot_*_all.svg

  evaluation_cox_model_all_vax_c:
    run: r:latest analysis/stage4_model_evaluation.R all_vax_c
    needs:
    - stage1_define_eligible_population_all
    outputs:
      moderately_sensitive:
        performance_measure_CSV: output/review/model/performance_measures_*_all_vax_c.csv
        performance_measure_HTML: output/review/model/performance_measures_*_all_vax_c.html
        survival_plot: output/review/model/survival_plot_*_all_vax_c.svg

  evaluation_cox_model_vaccinated:
    run: r:latest analysis/stage4_model_evaluation.R vaccinated
    needs:
    - stage1_define_eligible_population_vaccinated
    outputs:
      moderately_sensitive:
        performance_measure_CSV: output/review/model/performance_measures_*_vaccinated.csv
        performance_measure_HTML: output/review/model/performance_measures_*_vaccinated.html
        survival_plot: output/review/model/survival_plot_*_vaccinated.svg

  evaluation_cox_model_all_vax_td:
    run: r:latest analysis/stage4_model_evaluation.R all_vax_td
    needs:
    - stage1_define_eligible_population_all
    outputs:
      moderately_sensitive:
        performance_measure_CSV: output/review/model/performance_measures_*_all_vax_td.csv
        performance_measure_HTML: output/review/model/performance_measures_*_all_vax_td.html
        survival_plot: output/review/model/survival_plot_*_all_vax_td.svg

  evaluation_cox_model_infected:
    run: r:latest analysis/stage4_model_evaluation.R infected
    needs:
    - stage1_define_eligible_population_infected
    outputs:
      moderately_sensitive:
        performance_measure_CSV: output/review/model/performance_measures_*_infected.csv
        performance_measure_HTML: output/review/model/performance_measures_*_infected.html
        survival_plot: output/review/model/survival_plot_*_infected.svg

  ## Validation Cox Model - all 

  validation_cox_model_all:
    run: r:latest analysis/stage5_model_validation_iecv.R all
    needs:
    - stage1_define_eligible_population_all
    outputs:
      moderately_sensitive:
        val_performance_measure_CSV: output/review/model/val_performance_measures_all.csv
        val_performance_measure_html: output/review/model/val_performance_measures_all.html
        val_cal_plot: output/review/model/val_cal_plot_*_all.svg
        val_re_cal_plot: output/review/model/val_re_cal_plot_*_all.svg

  ## Validation Cox Model - all_vax_c 

  validation_cox_model_all_vax_c:
    run: r:latest analysis/stage5_model_validation_iecv.R all_vax_c
    needs:
    - stage1_define_eligible_population_all
    outputs:
      moderately_sensitive:
        val_performance_measure_CSV: output/review/model/val_performance_measures_all_vax_c.csv
        val_performance_measure_html: output/review/model/val_performance_measures_all_vax_c.html
        val_cal_plot: output/review/model/val_cal_plot_*_all_vax_c.svg
        val_re_cal_plot: output/review/model/val_re_cal_plot_*_all_vax_c.svg

  ## Validation Cox Model - vaccinated 

  validation_cox_model_vaccinated:
    run: r:latest analysis/stage5_model_validation_iecv.R vaccinated
    needs:
    - stage1_define_eligible_population_vaccinated
    outputs:
      moderately_sensitive:
        val_performance_measure_CSV: output/review/model/val_performance_measures_vaccinated.csv
        val_performance_measure_html: output/review/model/val_performance_measures_vaccinated.html
        val_cal_plot: output/review/model/val_cal_plot_*_vaccinated.svg
        val_re_cal_plot: output/review/model/val_re_cal_plot_*_vaccinated.svg

  ## Validation Cox Model - all_vax_td 

  validation_cox_model_all_vax_td:
    run: r:latest analysis/stage5_model_validation_iecv.R all_vax_td
    needs:
    - stage1_define_eligible_population_all
    outputs:
      moderately_sensitive:
        val_performance_measure_CSV: output/review/model/val_performance_measures_all_vax_td.csv
        val_performance_measure_html: output/review/model/val_performance_measures_all_vax_td.html
        val_cal_plot: output/review/model/val_cal_plot_*_all_vax_td.svg
        val_re_cal_plot: output/review/model/val_re_cal_plot_*_all_vax_td.svg

  ## Validation Cox Model - infected 

  validation_cox_model_infected:
    run: r:latest analysis/stage5_model_validation_iecv.R infected
    needs:
    - stage1_define_eligible_population_infected
    outputs:
      moderately_sensitive:
        val_performance_measure_CSV: output/review/model/val_performance_measures_infected.csv
        val_performance_measure_html: output/review/model/val_performance_measures_infected.html
        val_cal_plot: output/review/model/val_cal_plot_*_infected.svg
        val_re_cal_plot: output/review/model/val_re_cal_plot_*_infected.svg

State

State is inferred from the related Jobs.

Status: Succeeded

Timings

Timings set to UTC timezone.

  • Created:
  • Started:
  • Finished:
  • Runtime: 127:07:27

Config

  • Backend:
    TPP
  • Workspace:
    long-covid-risk-factors
  • Branch:
    main
  • Creator:
    yinghuiwei
  • Force run dependencies:
    False
  • Git Commit Hash:
    b4a1100
  • Requested actions:
    • development_cox_model_all
    • development_cox_model_all_vax_c
    • development_cox_model_vaccinated
    • development_cox_model_all_vax_td
    • development_cox_model_infected