Skip to content

Job request: 5488

Organisation:
University of Manchester
Workspace:
brit_cc_v2
ID:
xs4mnedlbkfg5fxu
View repo View project.yaml

This page shows the technical details of what happened when authorised researcher Ya-Ting Yang requested one or more actions to be run against real patient data in the project, within a secure environment.

By cross-referencing the indicated Requested Actions with the Pipeline section below, you can infer what security level various outputs were written to. Outputs marked as highly_sensitive can never be viewed directly by a researcher; they can only request that code runs against them. Outputs marked as moderately_sensitive can be viewed by an approved researcher by logging into a highly secure environment. Only outputs marked as moderately_sensitive can be requested for release to the public, via a controlled output review service.

Jobs

Pipeline

Show project.yaml 
version: '3.0'

expectations:
  population_size: 1000

actions:
  
  generate_study_population:
    run: cohortextractor:latest generate_cohort --study-definition study_definition_general_population
    outputs:
      highly_sensitive:
        cohort: output/input_general_population.csv

  generate_study_population_covid_primarycare:
    run: cohortextractor:latest generate_cohort --study-definition study_definition_covid_primarycare
    outputs:
      highly_sensitive:
        cohort: output/input_covid_primarycare.csv
  
  generate_study_population_covid_SGSS:
    run: cohortextractor:latest generate_cohort --study-definition study_definition_covid_SGSS
    outputs:
      highly_sensitive:
        cohort: output/input_covid_SGSS.csv
  
  generate_study_population_covid_admission:
    run: cohortextractor:latest generate_cohort --study-definition study_definition_covid_admission
    outputs:
      highly_sensitive:
        cohort: output/input_covid_admission.csv
  
  # generate_study_population_covid_icu:
  #   run: cohortextractor:latest generate_cohort --study-definition study_definition_covid_icu
  #   outputs:
  #     highly_sensitive:
  #       cohort: output/input_covid_icu.csv

  generate_study_population_covid_death_ons:
    run: cohortextractor:latest generate_cohort --study-definition study_definition_covid_death_ons
    outputs:
      highly_sensitive:
        cohort: output/input_covid_death_ons.csv
  
  generate_study_population_covid_death_cpns:
    run: cohortextractor:latest generate_cohort --study-definition study_definition_covid_death_cpns
    outputs:
      highly_sensitive:
        cohort: output/input_covid_death_cpns.csv
  
  process:
    run: r:latest analysis/process.R
    needs: [generate_study_population_covid_primarycare, generate_study_population_covid_SGSS,generate_study_population_covid_admission,generate_study_population_covid_death_ons,generate_study_population_covid_death_cpns]
    outputs:
      highly_sensitive:
        cohort1.1: output/case_covid_infection.csv
        cohort1.2: output/control_covid_infection.csv 
        #cohort1.3: output/case_covid_infection_*.csv # for monthly matching
        cohort2: output/case_covid_admission.csv
        cohort3: output/case_covid_icu_death.csv

  check_input: # check above 
    run: r:latest -e 'rmarkdown::render("analysis/check_input.Rmd", knit_root_dir = "/workspace", output_dir = "output")'
    needs: [process,generate_study_population, generate_study_population_covid_primarycare, generate_study_population_covid_SGSS,generate_study_population_covid_admission,generate_study_population_covid_death_ons,generate_study_population_covid_death_cpns]
    outputs:
      moderately_sensitive:
        html: output/check_input.html
        
  matching_1:
    run: python:latest python analysis/matching_outcome_1.py
    needs: [generate_study_population, process]
    outputs:
      moderately_sensitive: 
        matching_report1: output/matching_report_general_population_infection.txt
      highly_sensitive: 
        combined1: output/matched_combined_general_population_infection.csv

  matching_2:
    run: python:latest python analysis/matching_outcome_2.py
    needs: [process]
    outputs:
      moderately_sensitive: 
        matching_report2: output/matching_report_infection_hosp.txt
      highly_sensitive: 
        combined2: output/matched_combined_infection_hosp.csv

  matching_3:
    run: python:latest python analysis/matching_outcome_3.py
    needs: [process]
    outputs:
      moderately_sensitive: 
        matching_report3: output/matching_report_hosp_icu_death.txt 
      highly_sensitive: 
        combined3: output/matched_combined_hosp_icu_death.csv

  ## seperate matched by case and contorl
  pre_extract_variables_outcome_3:
    run: r:latest analysis/pre_extract_variables_outcome_3.R
    needs: [matching_3]
    outputs:
      highly_sensitive:
        cohort1: output/matched_outcome_3_case.csv
        cohort2: output/matched_outcome_3_control.csv
  
  extract_variables_outcome_3_case: 
    run: cohortextractor:latest generate_cohort --study-definition study_definition_outcome_3_case
    needs: [pre_extract_variables_outcome_3]
    outputs:
      highly_sensitive:
        cohort: output/input_outcome_3_case.csv


  extract_variables_outcome_1:
    run: cohortextractor:latest generate_cohort --study-definition study_definition_outcome_1
    needs: [matching_1]
    outputs:
      highly_sensitive:
        cohort: output/input_outcome_1.csv
  
  extract_variables_outcome_2:
    run: cohortextractor:latest generate_cohort --study-definition study_definition_outcome_2
    needs: [matching_2]
    outputs:
      highly_sensitive:
        cohort: output/input_outcome_2.csv

  # extract_variables_outcome_3: 
  #   run: cohortextractor:latest generate_cohort --study-definition study_definition_outcome_3
  #   needs: [matching_3]
  #   outputs:
  #     highly_sensitive:
  #       cohort: output/input_outcome_3.csv

  # variables:
  #   run: r:latest analysis/variables.R
  #   needs: [matching_1,matching_2,matching_3,extract_variables_outcome_1,extract_variables_outcome_2,extract_variables_outcome_3]
  #   outputs:
  #     highly_sensitive:
  #       cohort1: output/matched_outcome1.csv
  #       cohort2: output/matched_outcome2.csv
  #       cohort3: output/matched_outcome3.csv


  # check_matching:
  #   run: r:latest -e 'rmarkdown::render("analysis/check_matching.Rmd", knit_root_dir = "/workspace", output_dir="/workspace/output")'
  #   needs: [variables]
  #   outputs:
  #     moderately_sensitive:
  #       html: output/check_matching.html
  

 

# ## monthly matching datasets
#   generate_study_population_month:
#     run: cohortextractor:latest generate_cohort --study-definition study_definition_general_population --index-date-range "2020-02-01 to 2021-12-01 by month" --skip-existing --output-dir=output/measures --output-format=csv.gz
#     outputs:
#       highly_sensitive:
#         cohort: output/measures/input_general_population_*.csv.gz
  
#   exclusion:
#     run: r:latest analysis/exclusion.R
#     needs: [process, generate_study_population_month]
#     outputs:
#        highly_sensitive:
#         cohort4: output/control_general_population_*.csv
  

  generate_study_population_noCOVID:
    run: cohortextractor:latest generate_cohort --study-definition study_definition_noCOVID
    outputs:
      highly_sensitive:
        cohort: output/input_noCOVID.csv 
  
  check_input_noCOVID: 
    run: r:latest -e 'rmarkdown::render("analysis/check_input_noCOVID.Rmd", knit_root_dir = "/workspace", output_dir = "output")'
    needs: [generate_study_population_noCOVID]
    outputs:
      moderately_sensitive:
        html: output/check_input_noCOVID.html
  
  matching_1.2:
    run: python:latest python analysis/matching_outcome_1_noCOVID.py
    needs: [generate_study_population_noCOVID, process]
    outputs:
      moderately_sensitive: 
        matching_report1: output/matching_report_noCOVID_infection.txt
      highly_sensitive: 
        combined1: output/matched_combined_noCOVID_infection.csv
  
  extract_variables_outcome_1.2:
    run: cohortextractor:latest generate_cohort --study-definition study_definition_outcome_1_noCOVID
    needs: [matching_1.2]
    outputs:
      highly_sensitive:
        cohort: output/input_outcome_1_noCOVID.csv

Timeline

  • Created:

  • Started:

  • Finished:

  • Runtime: 00:38:00

These timestamps are generated and stored using the UTC timezone on the TPP backend.

Job information

Status
Succeeded
Backend
TPP
Workspace
brit_cc_v2
Requested by
Ya-Ting Yang
Branch
CC_AB_covid_V2
Force run dependencies
No
Git commit hash
64395c3
Requested actions
  • pre_extract_variables_outcome_3
  • extract_variables_outcome_3_case

Code comparison

Compare the code used in this Job Request