Skip to content

Job request: 6058

Organisation:
QMUL
Workspace:
bmi-and-metabolic-markers
ID:
45kbx2r7h22phyn5
View repo View project.yaml

This page shows the technical details of what happened when the authorised researcher Miriam Samuel requested one or more actions to be run against real patient data in the project, within a secure environment.

By cross-referencing the list of jobs with the pipeline section below, you can infer what security level various outputs were written to. Researchers can never directly view outputs marked as highly_sensitive ; they can only request that code runs against them. Outputs marked as moderately_sensitive can be viewed by an approved researcher by logging into a highly secure environment. Only outputs marked as moderately_sensitive can be requested for release to the public, via a controlled output review service.

Jobs

Pipeline

Show project.yaml 
version: '3.0'

expectations:
  population_size: 1000

actions:

### 1.  Extract the cohort

  ## use the Measure function to allow data for different time periods to be extracted in the same study population:  --index-date-range ""yyyy-mm-dd"" --output-format feather 
  generate_study_population_1:
    run: cohortextractor:latest generate_cohort --study-definition study_definition_all --output-dir=output/data --index-date-range "2015-03-01" --output-format feather
    outputs:
      highly_sensitive:
        cohort: output/data/input_all_2015-03-01.feather  

  generate_study_population_2:
    run: cohortextractor:latest generate_cohort --study-definition study_definition_all --output-dir=output/data --index-date-range "2016-03-01" --output-format feather
    outputs:
      highly_sensitive:
        cohort: output/data/input_all_2016-03-01.feather  

  generate_study_population_3:
    run: cohortextractor:latest generate_cohort --study-definition study_definition_all --output-dir=output/data --index-date-range "2017-03-01" --output-format feather
    outputs:
      highly_sensitive:
        cohort: output/data/input_all_2017-03-01.feather  
        
  generate_study_population_4:
    run: cohortextractor:latest generate_cohort --study-definition study_definition_all --output-dir=output/data --index-date-range "2018-03-01" --output-format feather 
    outputs:
      highly_sensitive:
        cohort: output/data/input_all_2018-03-01.feather  
        
        
  generate_study_population_5:
    run: cohortextractor:latest generate_cohort --study-definition study_definition_all --output-dir=output/data --index-date-range "2019-03-01" --output-format feather
    outputs:
      highly_sensitive:
        cohort: output/data/input_all_2019-03-01.feather        
        
  generate_study_population_6:
    run: cohortextractor:latest generate_cohort --study-definition study_definition_all --output-dir=output/data --index-date-range "2020-03-01" --output-format feather
    outputs:
      highly_sensitive:
        cohort: output/data/input_all_2020-03-01.feather  
   
  generate_study_population_7:
    run: cohortextractor:latest generate_cohort --study-definition study_definition_all --output-dir=output/data --index-date-range "2021-03-01" --output-format feather
    outputs:
      highly_sensitive:
        cohort: output/data/input_all_2021-03-01.feather  
 
 

  generate_study_population_ethnicity:
    run: cohortextractor:latest generate_cohort --study-definition study_definition_ethnicity --output-dir=output/data --output-format=feather 
    outputs:
      highly_sensitive:
        cohort: output/data/input_ethnicity.feather
        
      
        
  join_ethnicity:
    run: python:latest python analysis/join_ethnicity.py --output-dir=output/data --output-format feather
    needs: [generate_study_population_1, generate_study_population_2, generate_study_population_3, generate_study_population_4, generate_study_population_5, generate_study_population_6, generate_study_population_7, generate_study_population_ethnicity]
    outputs:
      highly_sensitive:
        cohort: output/data/input*.feather


###  Develop yearly BMI data sets to de-duplicate data and calculate median BMI each year.   Feather format to reduce data quantity.

  generate_BMI_2015_data:
    run: r:latest analysis/BMI_2015.R --output-dir=output/data --output-format feather
    needs: [join_ethnicity]
    outputs:
      highly_sensitive:
        cohort1: output/data/BMI_complete_median_2015.feather
        cohort2: output/data/BMI_complete_long_2015.feather


  generate_BMI_2016_data:
    run: r:latest analysis/BMI_2016.R --output-dir=output/data --output-format feather
    needs: [join_ethnicity]
    outputs:
      highly_sensitive:
        cohort1: output/data/BMI_complete_median_2016.feather
        cohort2: output/data/BMI_complete_long_2016.feather

  
  generate_BMI_2017_data:
    run: r:latest analysis/BMI_2017.R --output-dir=output/data --output-format feather
    needs: [join_ethnicity]
    outputs:
      highly_sensitive:
        cohort1: output/data/BMI_complete_median_2017.feather
        cohort2: output/data/BMI_complete_long_2017.feather
        
   
   
   
  generate_BMI_2019_data:
    run: r:latest analysis/BMI_2019.R --output-dir=output/data --output-format feather
    needs: [join_ethnicity]
    outputs:
      highly_sensitive:
        cohort1: output/data/BMI_complete_median_2019.feather
        cohort2: output/data/BMI_complete_long_2019.feather


  generate_BMI_2021_data:
    run: r:latest analysis/BMI_2021.R --output-dir=output/data --output-format feather
    needs: [join_ethnicity]
    outputs:
      highly_sensitive:
        cohort1: output/data/BMI_complete_median_2021.feather
        cohort2: output/data/BMI_complete_long_2021.feather


  generate_BMI_2018_data:
    run: r:latest analysis/BMI_2018.R --output-dir=output/data --output-format feather
    needs: [join_ethnicity]
    outputs:
      highly_sensitive:
        cohort1: output/data/BMI_complete_median_2018.feather
        cohort2: output/data/BMI_complete_long_2018.feather


  generate_BMI_2020_data:
    run: r:latest analysis/BMI_2020.R --output-dir=output/data --output-format feather
    needs: [join_ethnicity]
    outputs:
      highly_sensitive:
        cohort1: output/data/BMI_complete_median_2020.feather
        cohort2: output/data/BMI_complete_long_2020.feather


# Append yearly data sets to produce a complete data set for analysis of change in trends

  generate_complete_median_BMI_data:
    run: r:latest analysis/BMI_median_combine_datasets.R --output-dir=output/data --output-format feather
    needs: [generate_BMI_2015_data, generate_BMI_2016_data, generate_BMI_2017_data, generate_BMI_2018_data, generate_BMI_2019_data, generate_BMI_2020_data, generate_BMI_2021_data ]
    outputs:
      highly_sensitive:
        cohort1: output/data/BMI_complete_median.feather
        
 

##################################################################
###  HAD_BMI ANALYSIS
######################################################################
# who had_bmi measured:  univariate and multivariate analysis and tables
  generate_had_bmi_univariate_2019:
    run: r:latest analysis/had_bmi_regression_2019_univariate.R --output-dir=output/data 
    needs: [generate_complete_median_BMI_data]
    outputs:
      moderately_sensitive:
        table1: output/data/regression_had_bmi_2019.csv

  generate_had_bmi_univariate_2020:
    run: r:latest analysis/had_bmi_regression_2020_univariate.R --output-dir=output/data 
    needs: [generate_complete_median_BMI_data]
    outputs:
      moderately_sensitive:
        table1: output/data/regression_had_bmi_2020.csv
        
  generate_had_bmi_univariate_2021:
    run: r:latest analysis/had_bmi_regression_2021_univariate.R --output-dir=output/data 
    needs: [generate_complete_median_BMI_data]
    outputs:
      moderately_sensitive:
        table1: output/data/regression_had_bmi_2021.csv


  generate_had_bmi_multivariate_2019:
    run: r:latest analysis/had_bmi_regression_2019_multivariate.R --output-dir=output/data 
    needs: [generate_complete_median_BMI_data]
    outputs:
      moderately_sensitive:
        table1: output/data/multivariate_regression_had_bmi_2019.csv

  generate_had_bmi_multivariate_2020:
    run: r:latest analysis/had_bmi_regression_2020_multivariate.R --output-dir=output/data 
    needs: [generate_complete_median_BMI_data]
    outputs:
      moderately_sensitive:
        table1: output/data/multivariate_regression_had_bmi_2020.csv
        
  generate_had_bmi_multivariate_2021:
    run: r:latest analysis/had_bmi_regression_2021_multivariate.R --output-dir=output/data 
    needs: [generate_complete_median_BMI_data]
    outputs:
      moderately_sensitive:
        table1: output/data/multivariate_regression_had_bmi_2021.csv




 


 
 
  # generate summary stats of median BMI by exposures
  generate_median_summary_stats:
    run: r:latest analysis/BMI_median_summary_stats.R --output-dir=output/data 
    needs: [generate_complete_median_BMI_data]
    outputs:
      moderately_sensitive:
        table1: output/data/median_bmi_summary_table_demographic.csv
        table2: output/data/median_bmi_summary_table_covariates.csv

Timeline

  • Created:

  • Started:

  • Finished:

  • Runtime: 00:15:37

These timestamps are generated and stored using the UTC timezone on the TPP backend.

Job information

Status
Failed
Backend
TPP
Workspace
bmi-and-metabolic-markers
Requested by
Miriam Samuel
Branch
main
Force run dependencies
No
Git commit hash
e11ed18
Requested actions
  • generate_BMI_2020_data

Code comparison

Compare the code used in this Job Request