Skip to content

Job request: 1360

Organisation:
The London School of Hygiene & Tropical Medicine
Workspace:
carehomes-test-full
ID:
2qmu4bwxtw6gmvwh

This page shows the technical details of what happened when authorised researcher Emily Nightingale requested one or more actions to be run against real patient data in the project, within a secure environment.

By cross-referencing the indicated Requested Actions with the Pipeline section below, you can infer what security level various outputs were written to. Outputs marked as highly_sensitive can never be viewed directly by a researcher; they can only request that code runs against them. Outputs marked as moderately_sensitive can be viewed by an approved researcher by logging into a highly secure environment. Only outputs marked as moderately_sensitive can be requested for release to the public, via a controlled output review service.

Jobs

  • Action:
    generate_study_population
    Status:
    Status: Failed
    Job identifier:
    b76qcb2mejewieyg
  • Action:
    calc_coverage
    Status:
    Status: Failed
    Job identifier:
    g5v2javum7lnqnaj
  • Action:
    data_clean
    Status:
    Status: Failed
    Job identifier:
    xlt6receqm3ppfyy
  • Action:
    run_models
    Status:
    Status: Failed
    Job identifier:
    hno6v7tk6v6wnoxz
  • Action:
    data_setup
    Status:
    Status: Failed
    Job identifier:
    yojfltshpbgaxaod

Pipeline

Show project.yaml
version: '3.0'

expectations:
  population_size: 1000000

actions:
  generate_study_population:
    run: cohortextractor:latest generate_cohort --study-definition study_definition 
    outputs:
      highly_sensitive:
        cohort: input.csv

#  generate_coverage_population:
#    run: cohortextractor:latest generate_cohort --study-definition study_definition_coverage
#    outputs:
#      highly_sensitive:
#        cohort: input_coverage.csv

  calc_coverage:
    needs: [generate_study_population]
    run: r:latest analysis/calculate_tpp_coverage.R input.csv data/SAPE22DT15_mid_2019_msoa.csv
    outputs:
      moderately_sensitive:
        log: coverage_log.txt
        rds: tpp_msoa_coverage.rds
        csv: tpp_msoa_coverage.csv
        csv2: msoas_in_tpp.csv
        csv3: msoa_gt_100_cov.csv
        figure: total_vs_tpp_pop.png
        
  prelim:
    needs: [generate_study_population, calc_coverage]
    # last argument relates to MSOA TPP coverage >= X%
    run: r:latest analysis/prelim.R input.csv tpp_msoa_coverage.rds 80
    outputs:
      moderately_sensitive:
        log: prelim_check_log.txt

  data_clean:
    needs: [generate_study_population, calc_coverage]
    # last argument relates to MSOA TPP coverage >= X%
    run: r:latest analysis/data_clean.R input.csv tpp_msoa_coverage.rds 80
    outputs:
      moderately_sensitive:
        log: data_clean_log.txt
      highly_sensitive:
        input_clean: input_clean.rds
        
  data_check_figs:
    needs: [data_clean]
    run: r:latest analysis/data_check_figs.R input_clean.rds data/msoa_shp.rds
    outputs:
      moderately_sensitive:
        figure1: tpp_coverage_msoa.png
        figure2: tpp_coverage_carehomes.png
        figure3: tpp_coverage_map.pdf
        figure4: age_dist.png
        figure5: infection_death_delays.png
        figure6: hh_size_dist.png

  data_setup:
    needs: [data_clean]
    # last argument relates to carehome TPP coverage >= X%
    run: r:latest analysis/data_setup.R input_clean.rds 90
    outputs:
      moderately_sensitive:
        log: data_setup_log.txt
      highly_sensitive:
        comm_prev: community_prevalence.rds
        analysisdata: analysisdata.rds
        ch_linelist: ch_linelist.rds
        ch_agg_long: ch_agg_long.rds

  descriptive:
    needs: [data_clean, data_setup]
    run: r:latest analysis/descriptive.R 
    outputs:
      moderately_sensitive:
        report: descriptive.pdf
        log: log_descriptive.txt
        data: ch_gp_permsoa.csv

  run_models:
    needs: [data_setup]
    # 
    run: r:latest analysis/run_models.R analysisdata.rds community_prevalence.rds data/msoa_shp.rds 0.4
    outputs:
      moderately_sensitive:
        output: output_model_run.txt
        log: log_model_run.txt
       # figure: model_resids_map.pdf
      highly_sensitive:
        fit: fits.rds
        data: testdata.rds

  validate_models:
    needs: [run_models]
    run: r:latest analysis/validate_models.R fits.rds testdata.rds
    outputs:
      moderately_sensitive:
        output: output_model_val.txt
        report: test_pred_figs.pdf
        
  run_all:
    needs: [validate_models, descriptive]
    # In order to be valid this action needs to define a run commmand and
    # some output. We don't really care what these are but the below seems to
    # do the trick.
    run: cohortextractor:latest --version
    outputs:
      moderately_sensitive:
        whatever: project.yaml

Timeline

  • Created:

  • Started:

  • Finished:

  • Runtime: 00:09:39

These timestamps are generated and stored using the UTC timezone on the TPP backend.

Job information

Status
Failed
Backend
TPP
Requested by
Emily Nightingale
Branch
test
Force run dependencies
No
Git commit hash
27a779a
Requested actions
  • run_models

Code comparison

Compare the code used in this Job Request

  • No previous Job Request available for comparison