Skip to content

Job request: 6363

Organisation:
QMUL
Workspace:
bmi-and-metabolic-markers
ID:
bfopbckidk4xaloy
View repo View project.yaml

This page shows the technical details of what happened when the authorised researcher Miriam Samuel requested one or more actions to be run against real patient data within a secure environment.

By cross-referencing the list of jobs with the pipeline section below, you can infer what security level the outputs were written to.

The output security levels are:

  • highly_sensitive
    • Researchers can never directly view these outputs
    • Researchers can only request code is run against them
  • moderately_sensitive
    • Can be viewed by an approved researcher by logging into a highly secure environment
    • These are the only outputs that can be requested for public release via a controlled output review service.

Jobs

Pipeline

Show project.yaml 
version: '3.0'

expectations:
  population_size: 50000

actions:

### 1.  Extract the cohort

  ## use the Measure function to allow data for different time periods to be extracted in the same study population:  --index-date-range ""yyyy-mm-dd"" --output-format feather 
  generate_study_population_1:
    run: cohortextractor:latest generate_cohort --study-definition study_definition_all --output-dir=output/data --index-date-range "2015-03-01" --output-format feather
    outputs:
      highly_sensitive:
        cohort: output/data/input_all_2015-03-01.feather  

  generate_study_population_2:
    run: cohortextractor:latest generate_cohort --study-definition study_definition_all --output-dir=output/data --index-date-range "2016-03-01" --output-format feather
    outputs:
      highly_sensitive:
        cohort: output/data/input_all_2016-03-01.feather  

  generate_study_population_3:
    run: cohortextractor:latest generate_cohort --study-definition study_definition_all --output-dir=output/data --index-date-range "2017-03-01" --output-format feather
    outputs:
      highly_sensitive:
        cohort: output/data/input_all_2017-03-01.feather  
        
  generate_study_population_4:
    run: cohortextractor:latest generate_cohort --study-definition study_definition_all --output-dir=output/data --index-date-range "2018-03-01" --output-format feather 
    outputs:
      highly_sensitive:
        cohort: output/data/input_all_2018-03-01.feather  
        
        
  generate_study_population_5:
    run: cohortextractor:latest generate_cohort --study-definition study_definition_all --output-dir=output/data --index-date-range "2019-03-01" --output-format feather
    outputs:
      highly_sensitive:
        cohort: output/data/input_all_2019-03-01.feather        
        
  generate_study_population_6:
    run: cohortextractor:latest generate_cohort --study-definition study_definition_all --output-dir=output/data --index-date-range "2020-03-01" --output-format feather
    outputs:
      highly_sensitive:
        cohort: output/data/input_all_2020-03-01.feather  
   
  generate_study_population_7:
    run: cohortextractor:latest generate_cohort --study-definition study_definition_all --output-dir=output/data --index-date-range "2021-03-01" --output-format feather
    outputs:
      highly_sensitive:
        cohort: output/data/input_all_2021-03-01.feather  
 
 

  generate_study_population_ethnicity:
    run: cohortextractor:latest generate_cohort --study-definition study_definition_ethnicity --output-dir=output/data --output-format=feather 
    outputs:
      highly_sensitive:
        cohort: output/data/input_ethnicity.feather
        
      
        
  join_ethnicity:
    run: python:latest python analysis/join_ethnicity.py --output-dir=output/data --output-format feather
    needs: [generate_study_population_1, generate_study_population_2, generate_study_population_3, generate_study_population_4, generate_study_population_5, generate_study_population_6, generate_study_population_7, generate_study_population_ethnicity]
    outputs:
      highly_sensitive:
        cohort: output/data/input*.feather


###  Develop yearly BMI data sets to de-duplicate data and calculate median BMI each year.   Feather format to reduce data quantity.

  generate_BMI_2015_data:
    run: r:latest analysis/BMI_2015.R --output-dir=output/data --output-format feather
    needs: [join_ethnicity]
    outputs:
      highly_sensitive:
        cohort1: output/data/BMI_complete_median_2015.feather
        cohort2: output/data/BMI_complete_long_2015.feather


  generate_BMI_2016_data:
    run: r:latest analysis/BMI_2016.R --output-dir=output/data --output-format feather
    needs: [join_ethnicity]
    outputs:
      highly_sensitive:
        cohort1: output/data/BMI_complete_median_2016.feather
        cohort2: output/data/BMI_complete_long_2016.feather

  
  generate_BMI_2017_data:
    run: r:latest analysis/BMI_2017.R --output-dir=output/data --output-format feather
    needs: [join_ethnicity]
    outputs:
      highly_sensitive:
        cohort1: output/data/BMI_complete_median_2017.feather
        cohort2: output/data/BMI_complete_long_2017.feather
        
   
   
   
  generate_BMI_2019_data:
    run: r:latest analysis/BMI_2019.R --output-dir=output/data --output-format feather
    needs: [join_ethnicity]
    outputs:
      highly_sensitive:
        cohort1: output/data/BMI_complete_median_2019.feather
        cohort2: output/data/BMI_complete_long_2019.feather


  generate_BMI_2021_data:
    run: r:latest analysis/BMI_2021.R --output-dir=output/data --output-format feather
    needs: [join_ethnicity]
    outputs:
      highly_sensitive:
        cohort1: output/data/BMI_complete_median_2021.feather
        cohort2: output/data/BMI_complete_long_2021.feather


  generate_BMI_2018_data:
    run: r:latest analysis/BMI_2018.R --output-dir=output/data --output-format feather
    needs: [join_ethnicity]
    outputs:
      highly_sensitive:
        cohort1: output/data/BMI_complete_median_2018.feather
        cohort2: output/data/BMI_complete_long_2018.feather


  generate_BMI_2020_data:
    run: r:latest analysis/BMI_2020.R --output-dir=output/data --output-format feather
    needs: [join_ethnicity]
    outputs:
      highly_sensitive:
        cohort1: output/data/BMI_complete_median_2020.feather
        cohort2: output/data/BMI_complete_long_2020.feather


# Append yearly data sets to produce a complete data set for analysis of change in trends

  generate_complete_median_BMI_data:
    run: r:latest analysis/BMI_median_combine_datasets.R --output-dir=output/data --output-format feather
    needs: [generate_BMI_2015_data, generate_BMI_2016_data, generate_BMI_2017_data, generate_BMI_2018_data, generate_BMI_2019_data, generate_BMI_2020_data, generate_BMI_2021_data ]
    outputs:
      highly_sensitive:
        cohort1: output/data/BMI_complete_median.feather
        
#############################################################################################
### >> CREATE DATA SET WITH BMI AND WHOLE POPULATION DATA TO ASSESS ODDS OF HAVING BMI TEST
##############################################################################################
  generate_whole_pop_2019:
    run: r:latest analysis/create_BMI_all_2019.R --output-dir=output/data --output-format feather
    needs: [join_ethnicity, generate_BMI_2019_data]
    outputs:
      highly_sensitive:
        cohort1: output/data/BMI_all_2019.feather    

  generate_whole_pop_2020:
    run: r:latest analysis/create_BMI_all_2020.R --output-dir=output/data --output-format feather
    needs: [join_ethnicity, generate_BMI_2020_data, generate_complete_median_BMI_data]
    outputs:
      highly_sensitive:
        cohort1: output/data/BMI_all_2020.feather 

  generate_whole_pop_2021:
    run: r:latest analysis/create_BMI_all_2021.R --output-dir=output/data --output-format feather
    needs: [join_ethnicity, generate_BMI_2021_data, generate_complete_median_BMI_data]
    outputs:
      highly_sensitive:
        cohort1: output/data/BMI_all_2021.feather 
        
##################################################################
### >> HAD_BMI ANALYSIS
######################################################################
# who had_bmi measured:  univariate and multivariate analysis and tables
  generate_had_bmi_proportion_2019:
    run: r:latest analysis/had_bmi_proportions_2019.R --output-dir=output/data 
    needs: [generate_whole_pop_2019]
    outputs:
      moderately_sensitive:
        table1: output/data/proportion_had_bmi_2019.csv
  
  generate_had_bmi_proportion_2020:
    run: r:latest analysis/had_bmi_proportions_2020.R --output-dir=output/data 
    needs: [generate_whole_pop_2020]
    outputs:
      moderately_sensitive:
        table1: output/data/proportion_had_bmi_2020.csv
  
  generate_had_bmi_proportion_2021:
    run: r:latest analysis/had_bmi_proportions_2021.R --output-dir=output/data 
    needs: [generate_whole_pop_2021]
    outputs:
      moderately_sensitive:
        table1: output/data/proportion_had_bmi_2021.csv
 
#########################################################
## PROPORTION OBESE ANALYSIS

  generate_obese_proportion_2019:
    run: r:latest analysis/obese_proportions_2019.R --output-dir=output/data 
    needs: [generate_whole_pop_2019]
    outputs:
      moderately_sensitive:
        table1: output/data/proportion_obese_2019.csv


  generate_obese_proportion_2020:
    run: r:latest analysis/obese_proportions_2020.R --output-dir=output/data 
    needs: [generate_whole_pop_2020]
    outputs:
      moderately_sensitive:
        table1: output/data/proportion_obese_2020.csv

  generate_obese_proportion_2021:
    run: r:latest analysis/obese_proportions_2021.R --output-dir=output/data 
    needs: [generate_whole_pop_2021]
    outputs:
      moderately_sensitive:
        table1: output/data/proportion_obese_2021.csv






 ######################################################################################
 # >> Median BMI Analysis
 
 #######################################################################################

  
  
  # generate summary stats of median BMI by exposures
  generate_median_summary_stats:
    run: r:latest analysis/BMI_median_summary_stats.R --output-dir=output/data 
    needs: [generate_complete_median_BMI_data]
    outputs:
      moderately_sensitive:
        table1: output/data/median_bmi_summary_table_demographic.csv
        table2: output/data/median_bmi_summary_table_covariates.csv

Timeline

  • Created:

  • Started:

  • Finished:

  • Runtime: 00:02:53

These timestamps are generated and stored using the UTC timezone on the TPP backend.

Job request

Status
Succeeded
Backend
TPP
Workspace
bmi-and-metabolic-markers
Requested by
Miriam Samuel
Branch
main
Force run dependencies
No
Git commit hash
fbf0f44
Requested actions
  • generate_obese_proportion_2020

Code comparison

Compare the code used in this job request