Job request: 4784

Organisation:: University of Bristol
Workspace:: covid-ve-change-over-time--process-data-for-models
ID:: bgswtwflpwp3uk2w

This page shows the technical details of what happened when the authorised researcher Elsie Horne requested one or more actions to be run against real patient data within a secure environment.

By cross-referencing the list of jobs with the pipeline section below, you can infer what security level the outputs were written to.

The output security levels are:

highly_sensitive
- Researchers can never directly view these outputs
- Researchers can only request code is run against them
moderately_sensitive
- Can be viewed by an approved researcher by logging into a highly secure environment
- These are the only outputs that can be requested for public release via a controlled output review service.

Jobs

Action:

dummy_data_covs

Status:

Status: Succeeded

Job identifier:

loh2gtwm2ncjb72i
Action:

generate_study_population_covs

Status:

Status: Failed

Job identifier:

qq2lqolgqwnck6wm

Error:

nonzero_exit: Job exited with an error code

Pipeline

Show project.yaml

version: '3.0'

expectations:

  population_size: 100000

actions:

  ## #################################### 
  ## preliminaries 
  ## #################################### 

  design:
    run: r:latest analysis/design.R
    outputs:
      moderately_sensitive:
        study_dates_json: output/lib/study_parameters.json
        study_dates_rds: output/lib/study_parameters.rds
        jcvi_groups: output/lib/jcvi_groups.csv
        elig_dates: output/lib/elig_dates.csv
        regions: output/lib/regions.csv
        model_varlist: output/lib/model_varlist.rds

  ## #################################### 
  ## vax 
  ## #################################### 
  ## data for eligibility criteria and vaccines 
  ## generate dummy data for study_definition_vax 

  dummy_data_vax:
    run: r:latest analysis/vax/dummy_data_vax.R
    needs:
    - design
    outputs:
      moderately_sensitive:
        dummy_data_vax: analysis/vax/dummy_data_vax.feather

  ## study definition for eligiblity criteria and vaccine data 

  generate_study_population_vax:
    run: cohortextractor:latest generate_cohort --study-definition study_definition_vax
      --output-format feather
    dummy_data_file: analysis/vax/dummy_data_vax.feather
    needs:
    - design
    - dummy_data_vax
    outputs:
      highly_sensitive:
        cohort_vax: output/input_vax.feather

  ## process data from study_definition_vax 

  data_vax_process:
    run: r:latest analysis/vax/data_vax_process.R
    needs:
    - design
    - dummy_data_vax
    - generate_study_population_vax
    outputs:
      highly_sensitive:
        data_vax_covs: output/data/data_vax_covs.rds
        data_vax_dates: output/data/data_*_vax_dates.rds
      moderately_sensitive:
        data_properties: output/vax/tables/data_vax_processed_tabulate.txt

  ## apply eligiblity criteria from boxes a and b 

  data_eligible_ab:
    run: r:latest analysis/vax/data_eligible_ab.R
    needs:
    - design
    - data_vax_process
    outputs:
      highly_sensitive:
        data_eligible_a: output/data/data_eligible_a.rds
        data_eligible_b: output/data/data_eligible_b.rds
      moderately_sensitive:
        eligibility_count: output/lib/eligibility_count_ab.csv
        group_age_ranges: output/lib/group_age_ranges.csv

  ## #################################### 
  ## second_vax_period 
  ## #################################### 
  ## identify second vaccination time periods 
  ## create dataset for identifying second vaccination time periods 

  data_2nd_vax_dates:
    run: r:latest analysis/second_vax_period/data_2nd_vax_dates.R
    needs:
    - design
    - data_vax_process
    - data_eligible_ab
    outputs:
      highly_sensitive:
        data_vax_plot: output/second_vax_period/data/data_vax_plot.rds
        second_vax_period_dates_rds: output/lib/second_vax_period_dates.rds
      moderately_sensitive:
        second_vax_period_dates_csv: output/lib/second_vax_period_dates.csv
        start_dates: output/lib/start_dates.csv
        end_dates: output/lib/end_dates.csv

  ## identify and plot second vaccination time periods 

  plot_2nd_vax_dates:
    run: r:latest analysis/second_vax_period/plot_2nd_vax_dates.R
    needs:
    - design
    - data_eligible_ab
    - data_2nd_vax_dates
    outputs:
      moderately_sensitive:
        plots_by_region: output/second_vax_period/images/plot_by_region_*.png

  ## apply eligiblity criteria from boxes c and d 

  data_eligible_cd:
    run: r:latest analysis/second_vax_period/data_eligible_cd.R
    needs:
    - design
    - data_vax_process
    - data_eligible_ab
    - data_2nd_vax_dates
    outputs:
      highly_sensitive:
        data_eligible_c: output/data/data_eligible_c.rds
        data_eligible_d: output/data/data_eligible_d.rds

  ## #################################### 
  ## covs 
  ## #################################### 
  ## data for covariates 
  ## generate dummy data for study_definition_covs 

  dummy_data_covs:
    run: r:latest analysis/covs/dummy_data_covs.R
    needs:
    - design
    - dummy_data_vax
    - data_2nd_vax_dates
    outputs:
      moderately_sensitive:
        dummy_data_vax: analysis/covs/dummy_data_covs.feather

  ## study definition for covariates 

  generate_study_population_covs:
    run: cohortextractor:latest generate_cohort --study-definition study_definition_covs
      --output-format feather
    dummy_data_file: analysis/covs/dummy_data_covs.feather
    needs:
    - design
    - data_2nd_vax_dates
    - dummy_data_covs
    outputs:
      highly_sensitive:
        cohort_vax: output/input_covs.feather

  # ## process recurring variables as long data 

  # data_long_process:
  #   run: r:latest analysis/covs/data_long_process.R
  #   needs:
  #   - design
  #   - data_eligible_cd
  #   - generate_study_population_covs
  #   outputs:
  #     highly_sensitive:
  #       data_long_dates: output/data/data_long_*_dates.rds

  # ## #################################### 
  # ## comparisons 
  # ## #################################### 
  # ## process data, apply model and generate report for JCVI group 02 
  # ## process covariates data 

  # data_comparisons_process_02:
  #   run: r:latest analysis/comparisons/data_comparisons_process.R 02
  #   needs:
  #   - design
  #   - data_vax_process
  #   - data_2nd_vax_dates
  #   - data_eligible_cd
  #   - generate_study_population_covs
  #   - data_long_process
  #   outputs:
  #     highly_sensitive:
  #       data_comparisons: output/jcvi_group_02/data/data_comparisons.rds

  # ## process outcomes data 

  # data_outcomes_process_02:
  #   run: r:latest analysis/comparisons/data_outcomes_process.R 02
  #   needs:
  #   - design
  #   - generate_study_population_covs
  #   - data_long_process
  #   - data_comparisons_process_02
  #   outputs:
  #     highly_sensitive:
  #       data_outcomes: output/jcvi_group_02/data/data_outcomes.rds

  # ## check gap between outcomes for combining 

  # check_combine_outcomes_02:
  #   run: r:latest analysis/comparisons/check_combine_outcomes.R 02
  #   needs:
  #   - data_outcomes_process_02
  #   outputs:
  #     highly_sensitive:
  #       data_check_combine_outcomes: output/jcvi_group_02/data/check_combine_outcomes.rds
  #     moderately_sensitive:
  #       plot_check_combine_outcomes: output/jcvi_group_02/images/check_combine_outcomes.png
  #       table_check_combine_outcomes: output/jcvi_group_02/tables/check_combine_outcomes.csv

  # ## outcome = postest 
  # ## process tte data for postest 

  # data_tte_process_02_postest:
  #   run: r:latest analysis/comparisons/data_tte_process.R 02 postest
  #   needs:
  #   - data_comparisons_process_02
  #   - data_outcomes_process_02
  #   outputs:
  #     highly_sensitive:
  #       data_tte_brand_outcome: output/jcvi_group_02/data/data_tte_*_postest.rds

  # ## apply cox model for postest 

  # apply_model_cox_02_postest:
  #   run: r:latest analysis/comparisons/apply_model_cox.R 02 postest
  #   needs:
  #   - design
  #   - data_comparisons_process_02
  #   - data_tte_process_02_postest
  #   outputs:
  #     highly_sensitive:
  #       modelnumber: output/jcvi_group_02/models/*_postest_model*.rds
  #       model_tidy_rds: output/jcvi_group_02/models/*_postest_modelcox_tidy.rds
  #     moderately_sensitive:
  #       model_glance: output/jcvi_group_02/models/*_postest_modelcox_glance.csv
  #       model_tidy_csv: output/jcvi_group_02/models/*_postest_modelcox_tidy.csv

Timeline

Created: 3 years, 11 months ago 21 Dec 2021 19:23:28 UTC
Started: 3 years, 11 months ago 21 Dec 2021 19:23:51 UTC
Finished: 3 years, 11 months ago 21 Dec 2021 19:48:30 UTC
Runtime: 00:24:24

These timestamps are generated and stored using the UTC timezone on the TPP backend.

Job request

Status: Failed
Backend: TPP
Workspace: covid-ve-change-over-time--process-data-for-models
Requested by: Elsie Horne
Branch: process-data-for-models
Force run dependencies: No
Git commit hash: 5597cb3
Requested actions: dummy_data_covs

generate_study_population_covs

Code comparison

Compare the code used in this job request