Skip to content

Job request: 5018

Organisation:
University of Manchester
Workspace:
brit_cc_study
ID:
eer5zaxc6lyv6ynd
View repo View project.yaml

This page shows the technical details of what happened when the authorised researcher Ya-Ting Yang requested one or more actions to be run against real patient data in the project, within a secure environment.

By cross-referencing the list of jobs with the pipeline section below, you can infer what security level various outputs were written to. Researchers can never directly view outputs marked as highly_sensitive ; they can only request that code runs against them. Outputs marked as moderately_sensitive can be viewed by an approved researcher by logging into a highly secure environment. Only outputs marked as moderately_sensitive can be requested for release to the public, via a controlled output review service.

Jobs

  • Action:
    process
    Status:
    Status: Succeeded
    Job identifier:
    gkxlqoeq2nycbsjv
  • Action:
    matching
    Status:
    Status: Failed
    Job identifier:
    l7ouepuy3wicavmi

Pipeline

Show project.yaml 
version: '3.0'

expectations:
  population_size: 1000

actions:

  # generate_study_population:
  #   run: cohortextractor:latest generate_cohort --study-definition study_definition --index-date-range "2019-01-01 to today by month" --skip-existing --output-dir=output/measures --output-format=csv.gz
  #   outputs:
  #     highly_sensitive:
  #       cohort: output/measures/input_*.csv.gz

  # generate_study_population_elderly:
  #   run: cohortextractor:latest generate_cohort --study-definition study_definition_elderly 
  #     --output-format=csv.gz
  #   outputs:
  #     highly_sensitive:
  #       cohort: output/input_elderly.csv.gz

  # generate_measures:
  #   run: cohortextractor:latest generate_measures --study-definition study_definition --skip-existing --output-dir=output/measures
  #   needs: [generate_study_population]
  #   outputs:
  #     moderately_sensitive:
  #       measure_csv: output/measures/measure_*.csv
        
  # describe_elderly_agedis:
  #   run: r:latest analysis/tables/gen_csv_age_check.R
  #   needs: [generate_study_population_elderly]
  #   outputs:
  #     moderately_sensitive:
  #       agetable: output/age_quant.csv

  # describe:
  #   run: r:latest analysis/plot/overall_ab_prescribing.R
  #   needs: [generate_measures]
  #   outputs:
  #     moderately_sensitive:
  #       cohort: output/overall.png
  #       boxplot: output/overallbox.png

  # describe_percentile:
  #   run: r:latest analysis/plot/overall_ab_prescribing_2575percentile.R
  #   needs: [generate_measures]
  #   outputs:
  #     moderately_sensitive:
  #       percentile: output/overall_25th_75th_percentile.png

  # describe_starpu:
  #   run: r:latest analysis/plot/starpu_ab_prescribing.R
  #   needs: [generate_measures]
  #   outputs:
  #     moderately_sensitive:
  #       cohort: output/starpuline.png
  #       boxplot: output/starpubox.png
  
  # generate_notebook_starpu:
  #   run: jupyter:latest jupyter nbconvert /workspace/analysis/starpu.ipynb --execute --to html --output-dir=/workspace/output --ExecutePreprocessor.timeout=86400
  #   needs: [generate_measures]
  #   outputs:
  #     moderately_sensitive:
  #       notebook: output/starpu.html 
  #       figures: output/*
  #       tables: output/tables/*
  #       csvs: output/*/* # two possible subfolders
  #       text: output/text/*
  
  # describe_infection_ab_UTI:
  #   run: r:latest analysis/plot/infection_ab_UTI.R
  #   needs: [generate_study_population, generate_measures]
  #   outputs:
  #      moderately_sensitive:
  #       plot: output/UTI.png
  
  # generate_study_population:
  #   run: cohortextractor:latest generate_cohort --study-definition study_definition_covid_general_population --index-date-range "2020-02-01 to 2021-12-31 by month" --skip-existing --output-dir=output/measures --output-format=csv.gz
  #   outputs:
  #     highly_sensitive:
  #       cohort: output/measures/input_*.csv.gz
  
  generate_study_population:
    run: cohortextractor:latest generate_cohort --study-definition study_definition_general_population
    outputs:
      highly_sensitive:
        cohort: output/input_general_population.csv

  generate_study_population_covid_primarycare:
    run: cohortextractor:latest generate_cohort --study-definition study_definition_covid_primarycare
    outputs:
      highly_sensitive:
        cohort: output/input_covid_primarycare.csv
  
  generate_study_population_covid_SGSS:
    run: cohortextractor:latest generate_cohort --study-definition study_definition_covid_SGSS
    outputs:
      highly_sensitive:
        cohort: output/input_covid_SGSS.csv
  
  generate_study_population_covid_admission:
    run: cohortextractor:latest generate_cohort --study-definition study_definition_covid_admission
    outputs:
      highly_sensitive:
        cohort: output/input_covid_admission.csv
  
  # generate_study_population_covid_icu:
  #   run: cohortextractor:latest generate_cohort --study-definition study_definition_covid_icu
  #   outputs:
  #     highly_sensitive:
  #       cohort: output/input_covid_icu.csv

  generate_study_population_covid_death_ons:
    run: cohortextractor:latest generate_cohort --study-definition study_definition_covid_death_ons
    outputs:
      highly_sensitive:
        cohort: output/input_covid_death_ons.csv
  
  generate_study_population_covid_death_cpns:
    run: cohortextractor:latest generate_cohort --study-definition study_definition_covid_death_cpns
    outputs:
      highly_sensitive:
        cohort: output/input_covid_death_cpns.csv
  
  # exclusion:
  #   run: r:latest analysis/exclusion.R
  #   needs: [generate_study_population_covid_primarycare, generate_study_population_covid_SGSS,generate_study_population_covid_admission,generate_study_population_covid_icu,generate_study_population_covid_death_ons,generate_study_population_covid_death_cpns,generate_study_population]
  #   outputs:
  #      highly_sensitive:
  #       cohort1: output/case_covid_infection.csv
  #       cohort2: output/case_covid_admission.csv
  #       cohort3: output/case_covid_icu_death.csv
  #       cohort4: output/control_general_population_*.csv
  #       cohort5: output/case_covid_infection_*.csv

  process:
    run: r:latest analysis/process.R
    needs: [generate_study_population_covid_primarycare, generate_study_population_covid_SGSS,generate_study_population_covid_admission,generate_study_population_covid_death_ons,generate_study_population_covid_death_cpns]
    outputs:
      highly_sensitive:
        cohort1.1: output/case_covid_infection.csv
        cohort1.2: output/control_covid_infection.csv
        cohort2: output/case_covid_admission.csv
        cohort3: output/case_covid_icu_death.csv
        

  matching:
    run: python:latest python analysis/matching_case_control.py
    needs: [generate_study_population, process]
    outputs:
      moderately_sensitive: 
        matching_report1: output/matching_report_general_population_infection.txt
        matching_report2: output/matching_report_infection_hosp.txt
        matching_report3: output/matching_report_hosp_icu_death.txt 
      highly_sensitive: 
        combined1: output/matched_combined_general_population_infection.csv
        combined2: output/matched_combined_infection_hosp.csv
        combined3: output/matched_combined_hosp_icu_death.csv

Timeline

  • Created:

  • Started:

  • Finished:

  • Runtime: 105:25:23

These timestamps are generated and stored using the UTC timezone on the TPP backend.

Job information

Status
Failed
Backend
TPP
Workspace
brit_cc_study
Requested by
Ya-Ting Yang
Branch
case_control_AB_covid
Force run dependencies
No
Git commit hash
3caff41
Requested actions
  • process
  • matching

Code comparison

Compare the code used in this Job Request