Skip to content

Job request: 15715

Organisation:
King's College London
Workspace:
gout
ID:
ryst7dluumhxdptd
View repo View project.yaml

This page shows the technical details of what happened when authorised researcher Mark Russell requested one or more actions to be run against real patient data in the project, within a secure environment.

By cross-referencing the indicated Requested Actions with the Pipeline section below, you can infer what security level various outputs were written to. Outputs marked as highly_sensitive can never be viewed directly by a researcher; they can only request that code runs against them. Outputs marked as moderately_sensitive can be viewed by an approved researcher by logging into a highly secure environment. Only outputs marked as moderately_sensitive can be requested for release to the public, via a controlled output review service.

Jobs

Pipeline

Show project.yaml 
version: '3.0'

expectations:
  population_size: 100000

actions:

  generate_study_population:
    run: cohortextractor:latest generate_cohort --study-definition study_definition --output-format=csv.gz
    outputs:
      highly_sensitive:
        cohort: output/input.csv.gz

  generate_study_population_count:
    run: cohortextractor:latest generate_cohort --study-definition study_definition_count --output-format=csv.gz
    outputs:
      highly_sensitive:
        cohort: output/input_count.csv.gz

  summary_counts:
    run: stata-mp:latest analysis/002_summary_counts.do
    needs: [generate_study_population_count]
    outputs:
      highly_sensitive:
        log1: logs/summary_counts.log

  generate_study_population_allpts:
    run: cohortextractor:latest generate_cohort --study-definition study_definition_allpts --output-format=csv.gz
    outputs:
      highly_sensitive:
        cohort: output/input_allpts.csv.gz

  generate_study_population_2015:
    run: cohortextractor:latest generate_cohort --study-definition study_definition_year --index-date-range "2015-07-01" --output-dir=output/measures --output-format=csv.gz
    outputs:
      highly_sensitive:
        cohort: output/measures/input_year_2015-07-01.csv.gz

  generate_study_population_2016:
    run: cohortextractor:latest generate_cohort --study-definition study_definition_year --index-date-range "2016-07-01" --output-dir=output/measures --output-format=csv.gz
    outputs:
      highly_sensitive:
        cohort: output/measures/input_year_2016-07-01.csv.gz      

  generate_study_population_2017:
    run: cohortextractor:latest generate_cohort --study-definition study_definition_year --index-date-range "2017-07-01" --output-dir=output/measures --output-format=csv.gz
    outputs:
      highly_sensitive:
        cohort: output/measures/input_year_2017-07-01.csv.gz          

  generate_study_population_2018:
    run: cohortextractor:latest generate_cohort --study-definition study_definition_year --index-date-range "2018-07-01" --output-dir=output/measures --output-format=csv.gz
    outputs:
      highly_sensitive:
        cohort: output/measures/input_year_2018-07-01.csv.gz    
  
  generate_study_population_2019:
    run: cohortextractor:latest generate_cohort --study-definition study_definition_year --index-date-range "2019-07-01" --output-dir=output/measures --output-format=csv.gz
    outputs:
      highly_sensitive:
        cohort: output/measures/input_year_2019-07-01.csv.gz            

  generate_study_population_2020:
    run: cohortextractor:latest generate_cohort --study-definition study_definition_year --index-date-range "2020-07-01" --output-dir=output/measures --output-format=csv.gz
    outputs:
      highly_sensitive:
        cohort: output/measures/input_year_2020-07-01.csv.gz            

  generate_study_population_2021:
    run: cohortextractor:latest generate_cohort --study-definition study_definition_year --index-date-range "2021-07-01" --output-dir=output/measures --output-format=csv.gz
    outputs:
      highly_sensitive:
        cohort: output/measures/input_year_2021-07-01.csv.gz            

  generate_study_population_2022:
    run: cohortextractor:latest generate_cohort --study-definition study_definition_year --index-date-range "2022-07-01" --output-dir=output/measures --output-format=csv.gz
    outputs:
      highly_sensitive:
        cohort: output/measures/input_year_2022-07-01.csv.gz            
    
  generate_measures:
    run: cohortextractor:latest generate_measures --study-definition study_definition_year --output-dir=output/measures
    needs: [generate_study_population_2015, generate_study_population_2016, generate_study_population_2017, generate_study_population_2018, generate_study_population_2019, generate_study_population_2020, generate_study_population_2021, generate_study_population_2022]
    outputs:
      moderately_sensitive:
        measure_csv: output/measures/measure_*.csv           
  
  create_cohorts_allpts:
    run: stata-mp:latest analysis/001_define_covariates_allpts.do
    needs: [generate_study_population_allpts]
    outputs:
      highly_sensitive:
        log1: logs/cleaning_dataset_allpts.log 
        data1: output/data/file_gout_allpts.dta        

  create_cohorts:
    run: stata-mp:latest analysis/000_define_covariates.do
    needs: [generate_study_population, generate_measures]
    outputs:
      highly_sensitive:
        log1: logs/cleaning_dataset.log 
        data1: output/data/file_gout_all.dta
        data2: output/data/gout_prevalence_sex_long.dta
        data3: output/data/gout_incidence_sex_long.dta
        data4: output/data/gout_admissions_sex_long.dta

  run_baseline_tables_allpts:
    run: stata-mp:latest analysis/101_baseline_characteristics_allpts.do
    needs: [create_cohorts_allpts]
    outputs:
      moderately_sensitive:
        log1: logs/descriptive_tables_allpts.log      
        doc1: output/tables/baseline_allpts.csv      

  run_baseline_tables:
    run: stata-mp:latest analysis/100_baseline_characteristics.do
    needs: [create_cohorts]
    outputs:
      moderately_sensitive:
        log1: logs/descriptive_tables.log   
        doc1: output/tables/incidence_year_rounded.csv   
        doc2: output/tables/incidence_month_rounded.csv 
        doc3: output/tables/prevalance_year_rounded.csv   
        doc4: output/tables/incidence_admission_year_rounded.csv   
        doc6: output/tables/baseline_bydiagnosis.csv 
        doc7: output/tables/baseline_byyear.csv 
        doc8: output/tables/ult6m_byyear.csv   
        doc9: output/tables/ult6m_byregion.csv
        doc10: output/tables/urate6m_byyear.csv
        doc11: output/tables/urate6m_byregion.csv
        figure1: output/figures/incidence_year_rounded.svg
        figure2: output/figures/incidence_month_rounded.svg
        figure3: output/figures/prevalance_year_rounded.svg
        figure4: output/figures/incidence_admission_year_rounded.svg
  
  run_itsa_models:
    run: stata-mp:latest analysis/200_itsa_models.do
    needs: [create_cohorts]
    outputs:
      moderately_sensitive:
        log1: logs/itsa_models.log   
        figure1:  output/figures/ITSA_ult_newey.svg
        figure2:  output/figures/ITSA_360_newey.svg
      #  doc1: output/tables/gp_to_appt_ITSA_table.csv

  run_box_plots:
    run: stata-mp:latest analysis/300_box_plots.do
    needs: [create_cohorts]
    outputs:
      moderately_sensitive:
        log1: logs/box_plots.log
        figure 1: output/figures/regional_ult_overall.svg
        figure 2: output/figures/regional_ult_2019.svg 
        figure 3: output/figures/regional_ult_2020.svg       
        figure 4: output/figures/regional_ult_2021.svg  
        figure 5: output/figures/regional_ult_2022.svg 
        figure 6: output/figures/regional_ult_merged.svg

  # run_redacted_tables:
  #   run: stata-mp:latest analysis/400_redacted_tables.do
  #   needs: [create_cohorts]
  #   outputs:
  #     moderately_sensitive:
  #       log1: logs/redacted_tables.log   
  #       doc1: output/tables/table_1_rounded_bydiag.csv   
  #       doc2: output/tables/table_mean_bydiag_rounded.csv 
  #       doc3: output/tables/table_median_bydiag_rounded.csv   
  #       doc4: output/tables/table_median_bydiag_rounded_to21.csv   
  #       doc5: output/tables/ITSA_tables_appt_delay_rounded.csv   
  #       doc6: output/tables/ITSA_tables_csdmard_delay_rounded.csv 
  #       doc7: output/tables/drug_byyearanddisease_rounded.csv 
  #       doc8: output/tables/first_csdmard_rounded.csv   
  #       doc9: output/tables/drug_byyearandregion_rounded.csv
  #       doc10: output/tables/referral_byregion_rounded.csv
  #       doc11: output/tables/consultation_medium_rounded.csv
  #       doc12: output/tables/table_median_bydiag_rounded_to21_report.csv 
  #       doc13: output/tables/first_csdmard_rounded_report.csv  

  # run_redacted_tables_allpts:
  #   run: stata-mp:latest analysis/401_redacted_tables_allpts.do
  #   needs: [create_cohorts_allpts]
  #   outputs:
  #     moderately_sensitive:
  #       log1: logs/redacted_tables_allpts.log   
  #       doc1: output/tables/table_1_rounded_allpts.csv  

  # convert_image_formats:
  #   run: python:latest python analysis/convert_images.py --input_dir output/figures --output_dir output/figures
  #   needs: [run_baseline_tables, run_itsa_models, run_box_plots, run_redacted_tables]
  #   outputs:
  #     moderately_sensitive:
  #       figures: output/figures/*.png           

  # generate_notebook:
  #   run: jupyter:latest jupyter nbconvert /workspace/analysis/report.ipynb --execute --to html --template basic --output-dir=/workspace/output --ExecutePreprocessor.timeout=86400 --no-input
  #   needs: [convert_image_formats,run_baseline_tables, run_itsa_models, run_box_plots, run_redacted_tables]
  #   outputs:
  #     moderately_sensitive:
  #       notebook: output/report.html

Timeline

  • Created:

  • Started:

  • Finished:

  • Runtime: 00:16:58

These timestamps are generated and stored using the UTC timezone on the TPP backend.

Job information

Status
Succeeded
Backend
TPP
Workspace
gout
Requested by
Mark Russell
Branch
main
Force run dependencies
No
Git commit hash
7a82e5b
Requested actions
  • create_cohorts
  • run_baseline_tables

Code comparison

Compare the code used in this Job Request