Job request: 1360

View Repo View project.yaml

This page shows the technical details of what happened when authorised researcher esnightingale requested one or more actions to be run against real patient data in the Carehomes project, within a secure environment.

By cross-referencing the indicated Requested Actions with the Pipeline section below, you can infer what security level various outputs were written to. Outputs marked as highly_sensitive can never be viewed directly by a researcher; they can only request that code runs against them. Outputs marked as moderately_sensitive can be viewed by an approved researcher by logging into a highly secure environment. Only outputs marked as moderately_sensitive can be requested for release to the public, via a controlled output review service.

Jobs

ID Status Action
b76qcb2mejewieyg failed generate_study_population
g5v2javum7lnqnaj failed calc_coverage
xlt6receqm3ppfyy failed data_clean
yojfltshpbgaxaod failed data_setup
hno6v7tk6v6wnoxz failed run_models

Pipeline

Show Hide project.yaml
version: '3.0'

expectations:
  population_size: 1000000

actions:
  generate_study_population:
    run: cohortextractor:latest generate_cohort --study-definition study_definition 
    outputs:
      highly_sensitive:
        cohort: input.csv

#  generate_coverage_population:
#    run: cohortextractor:latest generate_cohort --study-definition study_definition_coverage
#    outputs:
#      highly_sensitive:
#        cohort: input_coverage.csv

  calc_coverage:
    needs: [generate_study_population]
    run: r:latest analysis/calculate_tpp_coverage.R input.csv data/SAPE22DT15_mid_2019_msoa.csv
    outputs:
      moderately_sensitive:
        log: coverage_log.txt
        rds: tpp_msoa_coverage.rds
        csv: tpp_msoa_coverage.csv
        csv2: msoas_in_tpp.csv
        csv3: msoa_gt_100_cov.csv
        figure: total_vs_tpp_pop.png
        
  prelim:
    needs: [generate_study_population, calc_coverage]
    # last argument relates to MSOA TPP coverage >= X%
    run: r:latest analysis/prelim.R input.csv tpp_msoa_coverage.rds 80
    outputs:
      moderately_sensitive:
        log: prelim_check_log.txt

  data_clean:
    needs: [generate_study_population, calc_coverage]
    # last argument relates to MSOA TPP coverage >= X%
    run: r:latest analysis/data_clean.R input.csv tpp_msoa_coverage.rds 80
    outputs:
      moderately_sensitive:
        log: data_clean_log.txt
      highly_sensitive:
        input_clean: input_clean.rds
        
  data_check_figs:
    needs: [data_clean]
    run: r:latest analysis/data_check_figs.R input_clean.rds data/msoa_shp.rds
    outputs:
      moderately_sensitive:
        figure1: tpp_coverage_msoa.png
        figure2: tpp_coverage_carehomes.png
        figure3: tpp_coverage_map.pdf
        figure4: age_dist.png
        figure5: infection_death_delays.png
        figure6: hh_size_dist.png

  data_setup:
    needs: [data_clean]
    # last argument relates to carehome TPP coverage >= X%
    run: r:latest analysis/data_setup.R input_clean.rds 90
    outputs:
      moderately_sensitive:
        log: data_setup_log.txt
      highly_sensitive:
        comm_prev: community_prevalence.rds
        analysisdata: analysisdata.rds
        ch_linelist: ch_linelist.rds
        ch_agg_long: ch_agg_long.rds

  descriptive:
    needs: [data_clean, data_setup]
    run: r:latest analysis/descriptive.R 
    outputs:
      moderately_sensitive:
        report: descriptive.pdf
        log: log_descriptive.txt
        data: ch_gp_permsoa.csv

  run_models:
    needs: [data_setup]
    # 
    run: r:latest analysis/run_models.R analysisdata.rds community_prevalence.rds data/msoa_shp.rds 0.4
    outputs:
      moderately_sensitive:
        output: output_model_run.txt
        log: log_model_run.txt
       # figure: model_resids_map.pdf
      highly_sensitive:
        fit: fits.rds
        data: testdata.rds

  validate_models:
    needs: [run_models]
    run: r:latest analysis/validate_models.R fits.rds testdata.rds
    outputs:
      moderately_sensitive:
        output: output_model_val.txt
        report: test_pred_figs.pdf
        
  run_all:
    needs: [validate_models, descriptive]
    # In order to be valid this action needs to define a run commmand and
    # some output. We don't really care what these are but the below seems to
    # do the trick.
    run: cohortextractor:latest --version
    outputs:
      moderately_sensitive:
        whatever: project.yaml

State

State is inferred from the related Jobs.

Status: Failed

Timings

Timings set to UTC timezone.

  • Created:
  • Started:
  • Finished:
  • Runtime: 00:09:39

Config

  • Backend:
    TPP
  • Workspace:
    carehomes-test-full
  • Branch:
    test
  • Creator:
    esnightingale
  • Force run dependencies:
    False
  • Git Commit Hash:
    27a779a
  • Requested actions:
    • run_models