Job request: 9692

View Repo View project.yaml

This page shows the technical details of what happened when authorised researcher Louis Fisher requested one or more actions to be run against real patient data in the 87 - Validation of the OpenSAFELY kidney codes project, within a secure environment.

By cross-referencing the indicated Requested Actions with the Pipeline section below, you can infer what security level various outputs were written to. Outputs marked as highly_sensitive can never be viewed directly by a researcher; they can only request that code runs against them. Outputs marked as moderately_sensitive can be viewed by an approved researcher by logging into a highly secure environment. Only outputs marked as moderately_sensitive can be requested for release to the public, via a controlled output review service.

Jobs

ID Status Action
st2paoywxpwrwqxr failed generate_study_population
z3t35f7brpw5d2zc failed generate_study_population_2

Pipeline

Show Hide project.yaml
version: "3.0"

expectations:
  population_size: 1000

actions:

  generate_study_population:
    run: cohortextractor:latest generate_cohort
      --study-definition study_definition
      --index-date-range "2019-01-01 to 2021-01-01 by month"
      --output-format csv.gz
    outputs:
      highly_sensitive:
        cohort: output/input*.csv.gz

  generate_study_population_2:
    run: cohortextractor:latest generate_cohort 
      --study-definition study_definition 
      --index-date-range "2021-02-01 to 2022-05-01 by month" 
      --output-format csv.gz
    outputs:
      highly_sensitive:
        cohort: output/inpu*.csv.gz

  generate_study_population_ethnicity_ukrr:
    run: cohortextractor:latest generate_cohort 
      --study-definition study_definition_ethnicity_ukrr
      --output-dir=output
      --output-format csv.gz
    outputs:
      highly_sensitive:
        cohort: output/input_ethnicity_ukrr.csv.gz

  join_cohorts:
    run: >
      cohort-joiner:v0.0.9
        --lhs output/input_20*.csv.gz
        --rhs output/input_ethnicity_ukrr.csv.gz
        --output-dir output/joined
    needs: [generate_study_population, generate_study_population_2, generate_study_population_ethnicity_ukrr]
    outputs:
      highly_sensitive:
        cohort: output/joined/input_20*.csv.gz


  generate_measures:
    run: cohortextractor:latest generate_measures 
      --study-definition study_definition
      --output-dir=output/joined
    needs: [join_cohorts]
    outputs:
      moderately_sensitive:
        measure_csv: output/joined/measure_*_rate.csv
        
  get_counts:
    run: python:latest python analysis/combine_operators.py
    needs: [join_cohorts]
    outputs:
      moderately_sensitive:
        counts: output/*_count.csv

  generate_plots:
    run: python:latest python analysis/plot_measures.py
    needs: [generate_measures]
    outputs:
      moderately_sensitive:
        counts: output/figures/plot_*.jpeg

  generate_plots_numeric_values:
    run: python:latest python analysis/plot_numeric_values.py
    needs: [join_cohorts]
    outputs:
      moderately_sensitive:
        figures: output/*dis*.jpeg


  # generate_notebook:
  #   run: jupyter:latest jupyter nbconvert /workspace/analysis/report.ipynb --execute --to html --template basic --output-dir=/workspace/output --ExecutePreprocessor.timeout=86400 --no-input
  #   needs: [generate_plots, get_counts]
  #   outputs:
  #     moderately_sensitive:
  #       notebook: output/report.html

  generate_ukrr_checks:
    run: r:latest -e 'rmarkdown::render("analysis/ukrr-analysis/ukrr-checks.Rmd", knit_root_dir = "/workspace", output_dir = "output/notebooks")'
    needs: [generate_study_population, generate_study_population_2, generate_study_population_ethnicity_ukrr]
    outputs:
      moderately_sensitive:
        report: output/notebooks/ukrr-checks.html

  generate_ukrr_report:
    run: r:latest -e 'rmarkdown::render("analysis/ukrr-analysis/ukrr-report.Rmd", knit_root_dir = "/workspace", output_dir = "output/notebooks")'
    needs: [join_cohorts]
    outputs:
      moderately_sensitive:
        report: output/notebooks/ukrr-report.html
       
  generate_ukrr_discrepancies:
    run: r:latest -e 'rmarkdown::render("analysis/ukrr-analysis/ukrr-discrepancies.Rmd", knit_root_dir = "/workspace", output_dir = "output/notebooks")'
    needs: [join_cohorts]
    outputs:
      moderately_sensitive:
        report: output/notebooks/ukrr-discrepancies.html
  
  generate_kidney_week:
    run: r:latest -e 'rmarkdown::render("analysis/ukrr-analysis/kidney_week.Rmd", knit_root_dir = "/workspace", output_dir = "output/notebooks")'
    needs: [join_cohorts]
    outputs:
      moderately_sensitive:
        report: output/notebooks/kidney_week.html
  
  generate_frequency_plots:
    run: r:latest analysis/test_frequency.R
    needs: [join_cohorts]
    outputs:
      moderately_sensitive:
        figures: output/figures/months_with_*.png

State

State is inferred from the related Jobs.

Status: Failed

Timings

Timings set to UTC timezone.

  • Created:
  • Started:
  • Finished:
  • Runtime: 06:21:48

Config

  • Backend:
    TPP
  • Workspace:
    renal-short-data-report
  • Branch:
    main
  • Creator:
    LFISHER7
  • Force run dependencies:
    False
  • Git Commit Hash:
    96aa124
  • Requested actions:
    • generate_study_population
    • generate_study_population_2