Job request: 11141

Organisation:: The UK Renal Registry
Workspace:: renal-short-data-report
ID:: sgvc6musetqcjdn2

This page shows the technical details of what happened when the authorised researcher Louis Fisher requested one or more actions to be run against real patient data within a secure environment.

By cross-referencing the list of jobs with the pipeline section below, you can infer what security level the outputs were written to.

The output security levels are:

highly_sensitive
- Researchers can never directly view these outputs
- Researchers can only request code is run against them
moderately_sensitive
- Can be viewed by an approved researcher by logging into a highly secure environment
- These are the only outputs that can be requested for public release via a controlled output review service.

Jobs

Action:

generate_study_population_calculators

Status:

Status: Succeeded

Job identifier:

yvz72xl5mwkzei5m
Action:

calculate_calculators

Status:

Status: Failed

Job identifier:

rt6bfvjbzvmccwsi

Error:

nonzero_exit: Job exited with error code 1
Action:

generate_notebook_crcl

Status:

Status: Failed

Job identifier:

5jfboyyxxyzix5ly

Error:

dependency_failed: Not starting as dependency failed

Pipeline

Show project.yaml

version: "3.0"

expectations:
  population_size: 1000

actions:

  generate_study_population_calculators:
    run: cohortextractor:latest generate_cohort
      --study-definition study_definition_calculators
      --output-format csv.gz
    outputs:
      highly_sensitive:
        cohort: output/input_calculators.csv.gz


  calculate_calculators:
    run: python:latest python analysis/scripts/calculate_calculators.py 
    needs: [generate_study_population_calculators]
    outputs:
      highly_sensitive:
        counts: output/input_calculators_calculated.csv.gz

  generate_notebook_crcl:
    run: jupyter:latest jupyter nbconvert /workspace/analysis/scripts/crcl_gault.ipynb --execute --to html --template basic --output-dir=/workspace/output --ExecutePreprocessor.timeout=86400 --no-input
    needs: [calculate_calculators]
    outputs:
      moderately_sensitive:
        notebook: output/crcl_gault.html

  generate_study_population:
    run: cohortextractor:latest generate_cohort
      --study-definition study_definition
      --index-date-range "2019-01-01 to 2021-01-01 by month"
      --output-format csv.gz
    outputs:
      highly_sensitive:
        cohort: output/input*.csv.gz

  generate_study_population_2:
   run: cohortextractor:latest generate_cohort 
     --study-definition study_definition 
     --index-date-range "2021-02-01 to 2022-07-01 by month" 
     --output-format csv.gz
   outputs:
     highly_sensitive:
       cohort: output/inpu*.csv.gz

  generate_study_population_ukrr_ethnicity:
    run: cohortextractor:latest generate_cohort 
      --study-definition study_definition_ukrr_ethnicity
      --output-dir=output
      --output-format csv.gz
    outputs:
      highly_sensitive:
        cohort: output/input_ukrr_ethnicity.csv.gz

  join_cohorts:
    run: >
      cohort-joiner:v0.0.46
        --lhs output/input_20*.csv.gz
        --rhs output/input_ukrr_ethnicity.csv.gz
        --output-dir output/joined
    needs: [generate_study_population, generate_study_population_2, generate_study_population_ukrr_ethnicity]
    outputs:
      highly_sensitive:
        cohort: output/joined/input_20*.csv.gz

  generate_study_population_paeds:
    run: cohortextractor:latest generate_cohort
      --study-definition study_definition_paeds
      --output-format csv.gz
    outputs:
      highly_sensitive:
        cohort: output/input_paeds.csv.gz


  generate_table_1:
    run: python:latest python analysis/scripts/table_1.py --study_def_paths="output/joined/input_*.csv.gz" --demographics="age_band,sex,region,imd,ethnicity"
    needs: [join_cohorts]
    outputs:
      moderately_sensitive:
        counts: output/table_1.csv
        at_risk: output/table_1_at_risk.csv


  get_counts:
    run: python:latest python analysis/scripts/combine_operators.py
    needs: [join_cohorts]
    outputs:
      moderately_sensitive:
        counts: output/*_count.csv

  generate_top_5_tables:
    run: python:latest python analysis/scripts/top_codes.py
    needs: [get_counts]
    outputs:
      moderately_sensitive:
        counts: output/top_5_code_table_*.csv

  generate_study_population_check_SC:
    run: cohortextractor:latest generate_cohort
      --study-definition study_definition_check_SC
      --output-format csv.gz
    outputs:
      highly_sensitive:
        cohort: output/input_check_SC.csv.gz

  generate_measures:
    run: cohortextractor:latest generate_measures 
      --study-definition study_definition
      --output-dir=output/joined
    needs: [join_cohorts]
    outputs:
      moderately_sensitive:
        measure_csv: output/joined/measure_*_rate.csv
        
  

  generate_plots:
    run: python:latest python analysis/scripts/plot_measures.py
    needs: [generate_measures]
    outputs:
      moderately_sensitive:
        counts: output/figures/plot_*.jpeg

  generate_plots_numeric_values:
    run: python:latest python analysis/scripts/plot_numeric_values.py
    needs: [join_cohorts]
    outputs:
      moderately_sensitive:
        figures: output/*dis*.jpeg

  generate_upset_plots:
    run: python:latest python analysis/scripts/ckd_discrepancies.py
    needs: [join_cohorts]
    outputs:
      moderately_sensitive:
        figures: output/ckd_staging_upset_*.png
        time: output/time_diff_ckd.jpeg

  # generate_notebook:
  #   run: jupyter:latest jupyter nbconvert /workspace/analysis/report.ipynb --execute --to html --template basic --output-dir=/workspace/output --ExecutePreprocessor.timeout=86400 --no-input
  #   needs: [generate_plots, get_counts]
  #   outputs:
  #     moderately_sensitive:
  #       notebook: output/report.html

  generate_ukrr_checks:
    run: r:latest -e 'rmarkdown::render("analysis/ukrr-analysis/ukrr-checks.Rmd", knit_root_dir = "/workspace", output_dir = "output/notebooks")'
    needs: [generate_study_population, generate_study_population_ukrr_ethnicity]
    outputs:
      moderately_sensitive:
        report: output/notebooks/ukrr-checks.html

  generate_ukrr_report:
    run: r:latest -e 'rmarkdown::render("analysis/ukrr-analysis/ukrr-report.Rmd", knit_root_dir = "/workspace", output_dir = "output/notebooks")'
    needs: [join_cohorts]
    outputs:
      moderately_sensitive:
        report: output/notebooks/ukrr-report.html
       
  generate_ukrr_discrepancies:
    run: r:latest -e 'rmarkdown::render("analysis/ukrr-analysis/ukrr-discrepancies.Rmd", knit_root_dir = "/workspace", output_dir = "output/notebooks")'
    needs: [join_cohorts]
    outputs:
      moderately_sensitive:
        report: output/notebooks/ukrr-discrepancies.html
  
  # generate_kidney_week:
  #   run: r:latest -e 'rmarkdown::render("analysis/ukrr-analysis/kidney_week.Rmd", knit_root_dir = "/workspace", output_dir = "output/notebooks")'
  #   needs: [join_cohorts]
  #   outputs:
  #     moderately_sensitive:
  #       report: output/notebooks/kidney_week.html
  
  generate_ukrr_tables_figures:
    run: r:latest -e 'rmarkdown::render("analysis/ukrr-analysis/ukrr-tables-figures.Rmd", knit_root_dir = "/workspace", output_dir = "output/notebooks")'
    needs: [join_cohorts]
    outputs:
      moderately_sensitive:
        report: output/notebooks/ukrr-tables-figures.html

  generate_ukrr_tables_figures_paeds:
    run: r:latest -e 'rmarkdown::render("analysis/ukrr-analysis/ukrr-tables-figures-paeds.Rmd", knit_root_dir = "/workspace", output_dir = "output/notebooks")'
    needs: [generate_study_population_paeds]
    outputs:
      moderately_sensitive:
        report: output/notebooks/ukrr-tables-figures-paeds.html
  
  generate_check_SC:
    run: r:latest -e 'rmarkdown::render("analysis/ukrr-analysis/check-SC.Rmd", knit_root_dir = "/workspace", output_dir = "output/notebooks")'
    needs: [generate_study_population_check_SC, generate_study_population_ukrr_ethnicity]
    outputs:
      moderately_sensitive:
        report: output/notebooks/check-SC.html
  
 # generate_frequency_plots:
 #   run: r:latest analysis/test_frequency.R
 #   needs: [join_cohorts_ethnicity]
 #   outputs:
 #     moderately_sensitive:
 #       figures: output/figures/months_with_*.png

Timeline

Created: 3 years, 2 months ago 26 Aug 2022 14:42:05 UTC
Started: 3 years, 2 months ago 26 Aug 2022 14:41:48 UTC
Finished: 3 years, 2 months ago 26 Aug 2022 17:08:29 UTC
Runtime: 02:26:02

These timestamps are generated and stored using the UTC timezone on the TPP backend.

Job request

Status: Failed
Backend: TPP
Workspace: renal-short-data-report
Requested by: Louis Fisher
Branch: main
Force run dependencies: No
Git commit hash: 60ba1b6
Requested actions: generate_study_population_calculators

calculate_calculators

generate_notebook_crcl

Code comparison

Compare the code used in this job request