Job request: 6358

Organisation:: QMUL
Workspace:: bmi-and-metabolic-markers
ID:: hazn2zy77czfstxr

This page shows the technical details of what happened when the authorised researcher Miriam Samuel requested one or more actions to be run against real patient data in the project, within a secure environment.

By cross-referencing the list of jobs with the pipeline section below, you can infer what security level various outputs were written to. Researchers can never directly view outputs marked as highly_sensitive ; they can only request that code runs against them. Outputs marked as moderately_sensitive can be viewed by an approved researcher by logging into a highly secure environment. Only outputs marked as moderately_sensitive can be requested for release to the public, via a controlled output review service.

Jobs

Action:

generate_obese_proportion_2019

Status:

Status: Failed

Job identifier:

5kkpfvuvbjuzstlh

Pipeline

Show project.yaml

version: '3.0'

expectations:
  population_size: 50000

actions:

### 1.  Extract the cohort

  ## use the Measure function to allow data for different time periods to be extracted in the same study population:  --index-date-range ""yyyy-mm-dd"" --output-format feather 
  generate_study_population_1:
    run: cohortextractor:latest generate_cohort --study-definition study_definition_all --output-dir=output/data --index-date-range "2015-03-01" --output-format feather
    outputs:
      highly_sensitive:
        cohort: output/data/input_all_2015-03-01.feather  

  generate_study_population_2:
    run: cohortextractor:latest generate_cohort --study-definition study_definition_all --output-dir=output/data --index-date-range "2016-03-01" --output-format feather
    outputs:
      highly_sensitive:
        cohort: output/data/input_all_2016-03-01.feather  

  generate_study_population_3:
    run: cohortextractor:latest generate_cohort --study-definition study_definition_all --output-dir=output/data --index-date-range "2017-03-01" --output-format feather
    outputs:
      highly_sensitive:
        cohort: output/data/input_all_2017-03-01.feather  
        
  generate_study_population_4:
    run: cohortextractor:latest generate_cohort --study-definition study_definition_all --output-dir=output/data --index-date-range "2018-03-01" --output-format feather 
    outputs:
      highly_sensitive:
        cohort: output/data/input_all_2018-03-01.feather  
        
        
  generate_study_population_5:
    run: cohortextractor:latest generate_cohort --study-definition study_definition_all --output-dir=output/data --index-date-range "2019-03-01" --output-format feather
    outputs:
      highly_sensitive:
        cohort: output/data/input_all_2019-03-01.feather        
        
  generate_study_population_6:
    run: cohortextractor:latest generate_cohort --study-definition study_definition_all --output-dir=output/data --index-date-range "2020-03-01" --output-format feather
    outputs:
      highly_sensitive:
        cohort: output/data/input_all_2020-03-01.feather  
   
  generate_study_population_7:
    run: cohortextractor:latest generate_cohort --study-definition study_definition_all --output-dir=output/data --index-date-range "2021-03-01" --output-format feather
    outputs:
      highly_sensitive:
        cohort: output/data/input_all_2021-03-01.feather  
 
 

  generate_study_population_ethnicity:
    run: cohortextractor:latest generate_cohort --study-definition study_definition_ethnicity --output-dir=output/data --output-format=feather 
    outputs:
      highly_sensitive:
        cohort: output/data/input_ethnicity.feather
        
      
        
  join_ethnicity:
    run: python:latest python analysis/join_ethnicity.py --output-dir=output/data --output-format feather
    needs: [generate_study_population_1, generate_study_population_2, generate_study_population_3, generate_study_population_4, generate_study_population_5, generate_study_population_6, generate_study_population_7, generate_study_population_ethnicity]
    outputs:
      highly_sensitive:
        cohort: output/data/input*.feather


###  Develop yearly BMI data sets to de-duplicate data and calculate median BMI each year.   Feather format to reduce data quantity.

  generate_BMI_2015_data:
    run: r:latest analysis/BMI_2015.R --output-dir=output/data --output-format feather
    needs: [join_ethnicity]
    outputs:
      highly_sensitive:
        cohort1: output/data/BMI_complete_median_2015.feather
        cohort2: output/data/BMI_complete_long_2015.feather


  generate_BMI_2016_data:
    run: r:latest analysis/BMI_2016.R --output-dir=output/data --output-format feather
    needs: [join_ethnicity]
    outputs:
      highly_sensitive:
        cohort1: output/data/BMI_complete_median_2016.feather
        cohort2: output/data/BMI_complete_long_2016.feather

  
  generate_BMI_2017_data:
    run: r:latest analysis/BMI_2017.R --output-dir=output/data --output-format feather
    needs: [join_ethnicity]
    outputs:
      highly_sensitive:
        cohort1: output/data/BMI_complete_median_2017.feather
        cohort2: output/data/BMI_complete_long_2017.feather
        
   
   
   
  generate_BMI_2019_data:
    run: r:latest analysis/BMI_2019.R --output-dir=output/data --output-format feather
    needs: [join_ethnicity]
    outputs:
      highly_sensitive:
        cohort1: output/data/BMI_complete_median_2019.feather
        cohort2: output/data/BMI_complete_long_2019.feather


  generate_BMI_2021_data:
    run: r:latest analysis/BMI_2021.R --output-dir=output/data --output-format feather
    needs: [join_ethnicity]
    outputs:
      highly_sensitive:
        cohort1: output/data/BMI_complete_median_2021.feather
        cohort2: output/data/BMI_complete_long_2021.feather


  generate_BMI_2018_data:
    run: r:latest analysis/BMI_2018.R --output-dir=output/data --output-format feather
    needs: [join_ethnicity]
    outputs:
      highly_sensitive:
        cohort1: output/data/BMI_complete_median_2018.feather
        cohort2: output/data/BMI_complete_long_2018.feather


  generate_BMI_2020_data:
    run: r:latest analysis/BMI_2020.R --output-dir=output/data --output-format feather
    needs: [join_ethnicity]
    outputs:
      highly_sensitive:
        cohort1: output/data/BMI_complete_median_2020.feather
        cohort2: output/data/BMI_complete_long_2020.feather


# Append yearly data sets to produce a complete data set for analysis of change in trends

  generate_complete_median_BMI_data:
    run: r:latest analysis/BMI_median_combine_datasets.R --output-dir=output/data --output-format feather
    needs: [generate_BMI_2015_data, generate_BMI_2016_data, generate_BMI_2017_data, generate_BMI_2018_data, generate_BMI_2019_data, generate_BMI_2020_data, generate_BMI_2021_data ]
    outputs:
      highly_sensitive:
        cohort1: output/data/BMI_complete_median.feather
        
#############################################################################################
### >> CREATE DATA SET WITH BMI AND WHOLE POPULATION DATA TO ASSESS ODDS OF HAVING BMI TEST
##############################################################################################
  generate_whole_pop_2019:
    run: r:latest analysis/create_BMI_all_2019.R --output-dir=output/data --output-format feather
    needs: [join_ethnicity, generate_BMI_2019_data]
    outputs:
      highly_sensitive:
        cohort1: output/data/BMI_all_2019.feather    

  generate_whole_pop_2020:
    run: r:latest analysis/create_BMI_all_2020.R --output-dir=output/data --output-format feather
    needs: [join_ethnicity, generate_BMI_2020_data, generate_complete_median_BMI_data]
    outputs:
      highly_sensitive:
        cohort1: output/data/BMI_all_2020.feather 

  generate_whole_pop_2021:
    run: r:latest analysis/create_BMI_all_2021.R --output-dir=output/data --output-format feather
    needs: [join_ethnicity, generate_BMI_2021_data, generate_complete_median_BMI_data]
    outputs:
      highly_sensitive:
        cohort1: output/data/BMI_all_2021.feather 
        
##################################################################
### >> HAD_BMI ANALYSIS
######################################################################
# who had_bmi measured:  univariate and multivariate analysis and tables
  generate_had_bmi_proportion_2019:
    run: r:latest analysis/had_bmi_proportions_2019.R --output-dir=output/data 
    needs: [generate_whole_pop_2019]
    outputs:
      moderately_sensitive:
        table1: output/data/proportion_had_bmi_2019.csv
  
  generate_had_bmi_proportion_2020:
    run: r:latest analysis/had_bmi_proportions_2020.R --output-dir=output/data 
    needs: [generate_whole_pop_2020]
    outputs:
      moderately_sensitive:
        table1: output/data/proportion_had_bmi_2020.csv
  
  generate_had_bmi_proportion_2021:
    run: r:latest analysis/had_bmi_proportions_2021.R --output-dir=output/data 
    needs: [generate_whole_pop_2021]
    outputs:
      moderately_sensitive:
        table1: output/data/proportion_had_bmi_2021.csv
 
#########################################################
## PROPORTION OBESE ANALYSIS

  generate_obese_proportion_2019:
    run: r:latest analysis/obese_proportions_2019.R --output-dir=output/data 
    needs: [generate_whole_pop_2019]
    outputs:
      moderately_sensitive:
        table1: output/data/proportion_obese_2019.csv






 ######################################################################################
 # >> Median BMI Analysis
 
 #######################################################################################

  
  
  # generate summary stats of median BMI by exposures
  generate_median_summary_stats:
    run: r:latest analysis/BMI_median_summary_stats.R --output-dir=output/data 
    needs: [generate_complete_median_BMI_data]
    outputs:
      moderately_sensitive:
        table1: output/data/median_bmi_summary_table_demographic.csv
        table2: output/data/median_bmi_summary_table_covariates.csv

Timeline

Created: 2 years, 7 months ago 27 Mar 2022 22:08:44 UTC
Finished: 2 years, 7 months ago 27 Mar 2022 22:15:32 UTC
Runtime:

These timestamps are generated and stored using the UTC timezone on the TPP backend.

Job information

Status: Failed
Backend: TPP
Workspace: bmi-and-metabolic-markers
Requested by: Miriam Samuel
Branch: main
Force run dependencies: No
Git commit hash: 73485be
Requested actions: generate_obese_proportion_2019

Code comparison

Compare the code used in this Job Request