Skip to content

Job request: 12590

Organisation:
The London School of Hygiene & Tropical Medicine
Workspace:
long_term_kidney_outcomes_after_covid
ID:
mnosexe322xo5kso
View repo View project.yaml

This page shows the technical details of what happened when the authorised researcher Viyaasan Mahalingasivam requested one or more actions to be run against real patient data in the project, within a secure environment.

By cross-referencing the list of jobs with the pipeline section below, you can infer what security level various outputs were written to. Researchers can never directly view outputs marked as highly_sensitive ; they can only request that code runs against them. Outputs marked as moderately_sensitive can be viewed by an approved researcher by logging into a highly secure environment. Only outputs marked as moderately_sensitive can be requested for release to the public, via a controlled output review service.

Jobs

Pipeline

Show project.yaml 
version: "3.0"

expectations:
  population_size: 50000

actions:

# Extract data required for matching for covid (all STPs)
  extract_covid_matching:
    run: cohortextractor:latest generate_cohort --study-definition study_definition_covid_matching
    outputs:
      highly_sensitive:
        cohort: output/input_covid_matching.csv

#Data management for covid_matching
  clean_covid_matching:
    run: stata-mp:latest analysis/covid_matching.do
    needs: [extract_covid_matching]
    outputs:
      highly_sensitive:
        analysis_dataset: output/covid_matching.csv
      moderately_sensitive:
        log: logs/covid_matching.log

# Extract data required for matching for contemporary (all STPs)
  extract_contemporary_matching:
    run: cohortextractor:latest generate_cohort --study-definition study_definition_contemporary_matching
    outputs:
      highly_sensitive:
        cohort: output/input_contemporary_matching.csv

#Data management for contemporary_matching
  clean_contemporary_matching:
    run: stata-mp:latest analysis/contemporary_matching.do
    needs: [extract_contemporary_matching]
    outputs:
      highly_sensitive:
        analysis_dataset: output/contemporary_matching.csv
      moderately_sensitive:
        log: logs/contemporary_matching.log

 # Check STPs
  check_stps:
    run: stata-mp:latest analysis/check_stps.do
    needs: [extract_covid_matching, clean_covid_matching, extract_contemporary_matching, clean_contemporary_matching]
    outputs:
      moderately_sensitive:
        log: logs/check_stps.log

# Extract data required for matching for covid (all STPs)
  extract_covid_matching_may2021:
    run: cohortextractor:latest generate_cohort --study-definition study_definition_covid_matching_may2021
    outputs:
      highly_sensitive:
        cohort: output/input_covid_matching_may2021.csv

#Data management for covid_matching
  clean_covid_matching_may2021:
    run: stata-mp:latest analysis/covid_matching_may2021.do
    needs: [extract_covid_matching_may2021]
    outputs:
      highly_sensitive:
        analysis_dataset: output/covid_matching_may2021.csv
      moderately_sensitive:
        log: logs/covid_matching_may2021.log

# Extract data required for matching for contemporary (all STPs)
  extract_contemporary_matching_may2021:
    run: cohortextractor:latest generate_cohort --study-definition study_definition_contemporary_matching_may2021
    outputs:
      highly_sensitive:
        cohort: output/input_contemporary_matching_may2021.csv

#Data management for contemporary_matching
  clean_contemporary_matching_may2021:
    run: stata-mp:latest analysis/contemporary_matching_may2021.do
    needs: [extract_contemporary_matching_may2021]
    outputs:
      highly_sensitive:
        analysis_dataset: output/contemporary_matching_may2021.csv
      moderately_sensitive:
        log: logs/contemporary_matching_may2021.log

# Split covid_matching and contemporary_matching by STP
  split_stps_may2021:
    run: stata-mp:latest analysis/split_stps_may2021.do
    needs: [extract_covid_matching_may2021, clean_covid_matching_may2021, extract_contemporary_matching_may2021, clean_contemporary_matching_may2021]
    outputs:
      highly_sensitive:
        covid_stp5_may2021: output/input_covid_matching_stp5_may2021.csv
        covid_stp6_may2021: output/input_covid_matching_stp6_may2021.csv
        covid_stp7_may2021: output/input_covid_matching_stp7_may2021.csv
        covid_stp8_may2021: output/input_covid_matching_stp8_may2021.csv
        covid_stp9_may2021: output/input_covid_matching_stp9_may2021.csv
        covid_stp10_may2021: output/input_covid_matching_stp10_may2021.csv
        covid_stp12_may2021: output/input_covid_matching_stp12_may2021.csv
        covid_stp13_may2021: output/input_covid_matching_stp13_may2021.csv
        covid_stp14_may2021: output/input_covid_matching_stp14_may2021.csv
        covid_stp15_may2021: output/input_covid_matching_stp15_may2021.csv
        covid_stp16_may2021: output/input_covid_matching_stp16_may2021.csv
        covid_stp17_may2021: output/input_covid_matching_stp17_may2021.csv
        covid_stp20_may2021: output/input_covid_matching_stp20_may2021.csv
        covid_stp21_may2021: output/input_covid_matching_stp21_may2021.csv
        covid_stp22_may2021: output/input_covid_matching_stp22_may2021.csv
        covid_stp23_may2021: output/input_covid_matching_stp23_may2021.csv
        covid_stp24_may2021: output/input_covid_matching_stp24_may2021.csv
        covid_stp25_may2021: output/input_covid_matching_stp25_may2021.csv
        covid_stp26_may2021: output/input_covid_matching_stp26_may2021.csv
        covid_stp27_may2021: output/input_covid_matching_stp27_may2021.csv
        covid_stp29_may2021: output/input_covid_matching_stp29_may2021.csv
        covid_stp33_may2021: output/input_covid_matching_stp33_may2021.csv
        covid_stp35_may2021: output/input_covid_matching_stp35_may2021.csv
        covid_stp36_may2021: output/input_covid_matching_stp36_may2021.csv
        covid_stp37_may2021: output/input_covid_matching_stp37_may2021.csv
        covid_stp40_may2021: output/input_covid_matching_stp40_may2021.csv
        covid_stp41_may2021: output/input_covid_matching_stp41_may2021.csv
        covid_stp42_may2021: output/input_covid_matching_stp42_may2021.csv
        covid_stp43_may2021: output/input_covid_matching_stp43_may2021.csv
        covid_stp44_may2021: output/input_covid_matching_stp44_may2021.csv
        covid_stp49_may2021: output/input_covid_matching_stp49_may2021.csv
        contemporary_stp5_may2021: output/input_contemporary_matching_stp5_may2021.csv
        contemporary_stp6_may2021: output/input_contemporary_matching_stp6_may2021.csv
        contemporary_stp7_may2021: output/input_contemporary_matching_stp7_may2021.csv
        contemporary_stp8_may2021: output/input_contemporary_matching_stp8_may2021.csv
        contemporary_stp9_may2021: output/input_contemporary_matching_stp9_may2021.csv
        contemporary_stp10_may2021: output/input_contemporary_matching_stp10_may2021.csv
        contemporary_stp12_may2021: output/input_contemporary_matching_stp12_may2021.csv
        contemporary_stp13_may2021: output/input_contemporary_matching_stp13_may2021.csv
        contemporary_stp14_may2021: output/input_contemporary_matching_stp14_may2021.csv
        contemporary_stp15_may2021: output/input_contemporary_matching_stp15_may2021.csv
        contemporary_stp16_may2021: output/input_contemporary_matching_stp16_may2021.csv
        contemporary_stp17_may2021: output/input_contemporary_matching_stp17_may2021.csv
        contemporary_stp20_may2021: output/input_contemporary_matching_stp20_may2021.csv
        contemporary_stp21_may2021: output/input_contemporary_matching_stp21_may2021.csv
        contemporary_stp22_may2021: output/input_contemporary_matching_stp22_may2021.csv
        contemporary_stp23_may2021: output/input_contemporary_matching_stp23_may2021.csv
        contemporary_stp24_may2021: output/input_contemporary_matching_stp24_may2021.csv
        contemporary_stp25_may2021: output/input_contemporary_matching_stp25_may2021.csv
        contemporary_stp26_may2021: output/input_contemporary_matching_stp26_may2021.csv
        contemporary_stp27_may2021: output/input_contemporary_matching_stp27_may2021.csv
        contemporary_stp29_may2021: output/input_contemporary_matching_stp29_may2021.csv
        contemporary_stp33_may2021: output/input_contemporary_matching_stp33_may2021.csv
        contemporary_stp35_may2021: output/input_contemporary_matching_stp35_may2021.csv
        contemporary_stp36_may2021: output/input_contemporary_matching_stp36_may2021.csv
        contemporary_stp37_may2021: output/input_contemporary_matching_stp37_may2021.csv
        contemporary_stp40_may2021: output/input_contemporary_matching_stp40_may2021.csv
        contemporary_stp41_may2021: output/input_contemporary_matching_stp41_may2021.csv
        contemporary_stp42_may2021: output/input_contemporary_matching_stp42_may2021.csv
        contemporary_stp43_may2021: output/input_contemporary_matching_stp43_may2021.csv
        contemporary_stp44_may2021: output/input_contemporary_matching_stp44_may2021.csv
        contemporary_stp49_may2021: output/input_contemporary_matching_stp49_may2021.csv
      moderately_sensitive:
        log: logs/split_stps_may2021.log

  match_contemporary_stp35_may2021:
    run: python:latest python analysis/match_contemporary_stp35_may2021.py
    needs: [extract_covid_matching_may2021, clean_covid_matching_may2021, extract_contemporary_matching_may2021, clean_contemporary_matching_may2021, split_stps_may2021]
    outputs:
      moderately_sensitive:
        matching_report_stp35_may2021: output/matching_report_contemporary_stp35_may2021.txt
      highly_sensitive:
        matched_matches_stp35_may2021: output/matched_matches_contemporary_stp35_may2021.csv
        matched_cases_stp35_may2021: output/matched_cases_contemporary_stp35_may2021.csv
        matched_all_stp35_may2021: output/matched_combined_contemporary_stp35_may2021.csv

Timeline

  • Created:

  • Started:

  • Finished:

  • Runtime: 15:54:01

These timestamps are generated and stored using the UTC timezone on the TPP backend.

Job information

Status
Succeeded
Backend
TPP
Workspace
long_term_kidney_outcomes_after_covid
Requested by
Viyaasan Mahalingasivam
Branch
main
Force run dependencies
No
Git commit hash
b928bfd
Requested actions
  • extract_covid_matching_may2021
  • clean_covid_matching_may2021
  • extract_contemporary_matching_may2021
  • clean_contemporary_matching_may2021
  • split_stps_may2021
  • match_contemporary_stp35_may2021

Code comparison

Compare the code used in this Job Request