Skip to content

Job request: 4172

Organisation:
Bennett Institute
Workspace:
sro-measures
ID:
d75usc5g7mzh2pal
View repo View project.yaml

This page shows the technical details of what happened when the authorised researcher Ben Butler-Cole requested one or more actions to be run against real patient data in the project, within a secure environment.

By cross-referencing the list of jobs with the pipeline section below, you can infer what security level various outputs were written to. Researchers can never directly view outputs marked as highly_sensitive ; they can only request that code runs against them. Outputs marked as moderately_sensitive can be viewed by an approved researcher by logging into a highly secure environment. Only outputs marked as moderately_sensitive can be requested for release to the public, via a controlled output review service.

Pipeline

Show project.yaml 
version: '3.0'

expectations:
  population_size: 1000
   
    
actions:

  generate_study_population:
    run: cohortextractor:latest generate_cohort --study-definition study_definition --index-date-range "2019-01-01 to 2021-04-01 by month" --output-dir=output --output-format=feather
    outputs:
      highly_sensitive:
        cohort: output/input_*.feather
        
  get_patient_count:
    run: python:latest python analysis/get_patients_counts.py
    needs: [generate_study_population]
    outputs:
      moderately_sensitive:
        text: output/patient_count.json
        

  generate_study_population_ethnicity:
    run: cohortextractor:latest generate_cohort --study-definition study_definition_ethnicity --output-dir=output --output-format=feather
    outputs:
      highly_sensitive:
        cohort: output/input_ethnicity.feather

  join_ethnicity:
    run: python:latest python analysis/join_ethnicity.py
    needs: [generate_study_population, generate_study_population_ethnicity]
    outputs:
      highly_sensitive:
        cohort: output/input*.feather

 
        
  get_practice_count:
    run: python:latest python analysis/get_practice_count.py
    needs: [join_ethnicity]
    outputs:
      moderately_sensitive:
        text: output/practice_count.json
        
  generate_measures:
    run: cohortextractor:latest generate_measures --study-definition study_definition --output-dir=output
    needs: [join_ethnicity]
    outputs:
        moderately_sensitive:
          measure_csv: output/measure_*.csv

  generate_measures_cleaned:
    run: python:latest python analysis/clean_measures.py
    needs: [generate_measures]
    outputs:
        moderately_sensitive:
          measure_csv: output/measure_cleaned_*.csv
          

  generate_measures_demographics:
    run: python:latest python analysis/calculate_measures.py
    needs: [join_ethnicity]
    outputs:
      moderately_sensitive:
        measure: output/combined_measure_*.csv
    
  
  generate_notebook:
    run: jupyter:latest jupyter nbconvert /workspace/notebooks/sentinel_measures.ipynb --execute --to html --template basic --output-dir=/workspace/output --ExecutePreprocessor.timeout=86400 --no-input
    needs: [generate_measures, get_practice_count, get_patient_count]
    outputs:
      moderately_sensitive:
        notebook: output/sentinel_measures.html
        subplots: output/sentinel_measures_subplots.png
  

  generate_notebook_demographics:
    run: jupyter:latest jupyter nbconvert /workspace/notebooks/sentinel_measures_demographics.ipynb --execute --to html --template basic --output-dir=/workspace/output --ExecutePreprocessor.timeout=86400 --no-input
    needs: [generate_measures, generate_measures_demographics]
    outputs:
      moderately_sensitive:
        notebook: output/sentinel_measures_demographics.html

  get_population_count:
    run: python:latest python analysis/population_counts.py
    needs: [join_ethnicity]
    outputs:
      moderately_sensitive:
        text: output/*_count.csv

  demographic_changes:
    run: python:latest python notebooks/demographic_change.py
    needs: [generate_measures_demographics]
    outputs:
      moderately_sensitive:
        csv: output/demographics_differences.csv
        csv_sorted: output/demographics_differences_sorted.csv
        

#   run_tests:
#     run: python:latest python -m pytest --junit-xml=output/pytest.xml --verbose
#     outputs:
#       moderately_sensitive:
#         log: output/pytest.xml

Timeline

  • Created:

  • Started:

  • Finished:

  • Runtime:

These timestamps are generated and stored using the UTC timezone on the TEST backend.

Job information

Status
Failed 
JobRequestError: All requested actions were already scheduled to run: __error__(FAILED), demographic_changes(SUCCEEDED), generate_measures(SUCCEEDED), generate_measures_cleaned(SUCCEEDED), generate_measures_demographics(SUCCEEDED), generate_notebook(SUCCEEDED), generate_notebook_demographics(SUCCEEDED), generate_study_population(SUCCEEDED), generate_study_population_ethnicity(SUCCEEDED), generate_study_population_practice_count(SUCCEEDED), get_patient_count(SUCCEEDED), get_population_count(SUCCEEDED), get_practice_count(SUCCEEDED), join_ethnicity(SUCCEEDED), run_all(SUCCEEDED)
Backend
TEST
Workspace
sro-measures
Requested by
Ben Butler-Cole
Branch
master
Force run dependencies
No
Git commit hash
d820b4c
Requested actions
  • run_all

Code comparison

Compare the code used in this Job Request