Skip to content

Job request: 11792

Organisation:
The London School of Hygiene & Tropical Medicine
Workspace:
long_term_kidney_outcomes_after_covid
ID:
534xueff5ohaei5n
View repo View project.yaml

This page shows the technical details of what happened when the authorised researcher Viyaasan Mahalingasivam requested one or more actions to be run against real patient data in the project, within a secure environment.

By cross-referencing the list of jobs with the pipeline section below, you can infer what security level various outputs were written to. Researchers can never directly view outputs marked as highly_sensitive ; they can only request that code runs against them. Outputs marked as moderately_sensitive can be viewed by an approved researcher by logging into a highly secure environment. Only outputs marked as moderately_sensitive can be requested for release to the public, via a controlled output review service.

Jobs

Pipeline

Show project.yaml 
version: "3.0"

expectations:
  population_size: 50000

actions:

# Extract data required for matching for covid (all STPs)
  extract_covid_matching:
    run: cohortextractor:latest generate_cohort --study-definition study_definition_covid_matching
    outputs:
      highly_sensitive:
        cohort: output/input_covid_matching.csv

#Data management for covid_matching
  clean_covid_matching:
    run: stata-mp:latest analysis/covid_matching.do
    needs: [extract_covid_matching]
    outputs:
      highly_sensitive:
        analysis_dataset: output/covid_matching.csv
      moderately_sensitive:
        log: logs/covid_matching.log

# Extract data required for matching for contemporary (all STPs)
  extract_contemporary_matching:
    run: cohortextractor:latest generate_cohort --study-definition study_definition_contemporary_matching
    outputs:
      highly_sensitive:
        cohort: output/input_contemporary_matching.csv

#Data management for contemporary_matching
  clean_contemporary_matching:
    run: stata-mp:latest analysis/contemporary_matching.do
    needs: [extract_contemporary_matching]
    outputs:
      highly_sensitive:
        analysis_dataset: output/contemporary_matching.csv
      moderately_sensitive:
        log: logs/contemporary_matching.log

 # Check STPs
  check_stps:
    run: stata-mp:latest analysis/check_stps.do
    needs: [extract_covid_matching, extract_contemporary_matching]
    outputs:
      moderately_sensitive:
        log: logs/check_stps.log

# Split covid_matching and contemporary_matching by STP
  split_stps:
    run: stata-mp:latest analysis/split_stps.do
    needs: [extract_covid_matching, extract_contemporary_matching]
    outputs:
      highly_sensitive:
        covid_stp5: output/input_covid_matching_stp5.csv
        covid_stp6: output/input_covid_matching_stp6.csv
        covid_stp7: output/input_covid_matching_stp7.csv
        covid_stp8: output/input_covid_matching_stp8.csv
        covid_stp9: output/input_covid_matching_stp9.csv
        covid_stp10: output/input_covid_matching_stp10.csv
        covid_stp12: output/input_covid_matching_stp12.csv
        covid_stp13: output/input_covid_matching_stp13.csv
        covid_stp14: output/input_covid_matching_stp14.csv
        covid_stp15: output/input_covid_matching_stp15.csv
        covid_stp16: output/input_covid_matching_stp16.csv
        covid_stp17: output/input_covid_matching_stp17.csv
        covid_stp20: output/input_covid_matching_stp20.csv
        covid_stp21: output/input_covid_matching_stp21.csv
        covid_stp22: output/input_covid_matching_stp22.csv
        covid_stp23: output/input_covid_matching_stp23.csv
        covid_stp24: output/input_covid_matching_stp24.csv
        covid_stp25: output/input_covid_matching_stp25.csv
        covid_stp26: output/input_covid_matching_stp26.csv
        covid_stp27: output/input_covid_matching_stp27.csv
        covid_stp29: output/input_covid_matching_stp29.csv
        covid_stp33: output/input_covid_matching_stp33.csv
        covid_stp35: output/input_covid_matching_stp35.csv
        covid_stp36: output/input_covid_matching_stp36.csv
        covid_stp37: output/input_covid_matching_stp37.csv
        covid_stp40: output/input_covid_matching_stp40.csv
        covid_stp41: output/input_covid_matching_stp41.csv
        covid_stp42: output/input_covid_matching_stp42.csv
        covid_stp43: output/input_covid_matching_stp43.csv
        covid_stp44: output/input_covid_matching_stp44.csv
        covid_stp49: output/input_covid_matching_stp49.csv
        contemporary_stp5: output/input_contemporary_matching_stp5.csv
        contemporary_stp6: output/input_contemporary_matching_stp6.csv
        contemporary_stp7: output/input_contemporary_matching_stp7.csv
        contemporary_stp8: output/input_contemporary_matching_stp8.csv
        contemporary_stp9: output/input_contemporary_matching_stp9.csv
        contemporary_stp10: output/input_contemporary_matching_stp10.csv
        contemporary_stp12: output/input_contemporary_matching_stp12.csv
        contemporary_stp13: output/input_contemporary_matching_stp13.csv
        contemporary_stp14: output/input_contemporary_matching_stp14.csv
        contemporary_stp15: output/input_contemporary_matching_stp15.csv
        contemporary_stp16: output/input_contemporary_matching_stp16.csv
        contemporary_stp17: output/input_contemporary_matching_stp17.csv
        contemporary_stp20: output/input_contemporary_matching_stp20.csv
        contemporary_stp21: output/input_contemporary_matching_stp21.csv
        contemporary_stp22: output/input_contemporary_matching_stp22.csv
        contemporary_stp23: output/input_contemporary_matching_stp23.csv
        contemporary_stp24: output/input_contemporary_matching_stp24.csv
        contemporary_stp25: output/input_contemporary_matching_stp25.csv
        contemporary_stp26: output/input_contemporary_matching_stp26.csv
        contemporary_stp27: output/input_contemporary_matching_stp27.csv
        contemporary_stp29: output/input_contemporary_matching_stp29.csv
        contemporary_stp33: output/input_contemporary_matching_stp33.csv
        contemporary_stp35: output/input_contemporary_matching_stp35.csv
        contemporary_stp36: output/input_contemporary_matching_stp36.csv
        contemporary_stp37: output/input_contemporary_matching_stp37.csv
        contemporary_stp40: output/input_contemporary_matching_stp40.csv
        contemporary_stp41: output/input_contemporary_matching_stp41.csv
        contemporary_stp42: output/input_contemporary_matching_stp42.csv
        contemporary_stp43: output/input_contemporary_matching_stp43.csv
        contemporary_stp44: output/input_contemporary_matching_stp44.csv
        contemporary_stp49: output/input_contemporary_matching_stp49.csv
      moderately_sensitive:
        log: logs/split_stps.log


# Match covid_matching_stp5 to contemporary_matching_stp5
  match_contemporary_stp5:
    run: python:latest python analysis/match_contemporary_stp5.py
    needs: [extract_covid_matching, clean_covid_matching, extract_contemporary_matching, clean_contemporary_matching, split_stps]
    outputs:
      moderately_sensitive:
        matching_report: output/matching_report_contemporary_stp5.txt
      highly_sensitive:
        matched_matches: output/matched_matches_contemporary_stp5.csv

Timeline

  • Created:

  • Started:

  • Finished:

  • Runtime: 02:47:18

These timestamps are generated and stored using the UTC timezone on the TPP backend.

Job information

Status
Failed
Backend
TPP
Workspace
long_term_kidney_outcomes_after_covid
Requested by
Viyaasan Mahalingasivam
Branch
main
Force run dependencies
No
Git commit hash
d479959
Requested actions
  • extract_covid_matching
  • clean_covid_matching
  • extract_contemporary_matching
  • clean_contemporary_matching
  • check_stps
  • split_stps
  • match_contemporary_stp5

Code comparison

Compare the code used in this Job Request