Job request: 19258

Organisation:: Bennett Institute
Workspace:: sro-measures-dashboard-updating
ID:: deac56e6rounl5ow

This page shows the technical details of what happened when the authorised researcher Rose Higgins requested one or more actions to be run against real patient data within a secure environment.

By cross-referencing the list of jobs with the pipeline section below, you can infer what security level the outputs were written to.

The output security levels are:

highly_sensitive
- Researchers can never directly view these outputs
- Researchers can only request code is run against them
moderately_sensitive
- Can be viewed by an approved researcher by logging into a highly secure environment
- These are the only outputs that can be requested for public release via a controlled output review service.

Jobs

Action:

run_tests

Status:

Succeeded

Job identifier:

jmsvrxe5crqcgung
Action:

generate_study_population_1

Status:

Succeeded

Job identifier:

ium6whgbmejibwpm
Action:

generate_study_population_2

Status:

Succeeded

Job identifier:

t227vigjurzod5y3
Action:

generate_study_population_3

Status:

Succeeded

Job identifier:

whupzwbdb7gmxxag
Action:

generate_study_population_4

Status:

Succeeded

Job identifier:

q2q7k77cjj57ag3u
Action:

generate_study_population_5

Status:

Failed

Job identifier:

cttcpsu4fcdhc2md

Error:

nonzero_exit: Job exited with an error: Something went wrong with the database, please contact tech support
Action:

generate_study_population_ethnicity

Status:

Succeeded

Job identifier:

ibupumul72rcnm3b
Action:

generate_study_population_population

Status:

Succeeded

Job identifier:

ogly3vg5syb2k3kr
Action:

join_ethnicity_population

Status:

Succeeded

Job identifier:

gbz77popg6fmul36
Action:

get_moved_count

Status:

Succeeded

Job identifier:

uuhzz3rruboapq2g
Action:

generate_notebook

Status:

Failed

Job identifier:

o5tvtmi5ownfmqi5

Error:

dependency_failed: Not starting as dependency failed
Action:

generate_measures

Status:

Failed

Job identifier:

5vy5svhlcowstscj

Error:

dependency_failed: Not starting as dependency failed
Action:

generate_measures_cleaned

Status:

Failed

Job identifier:

lt2cx2rfzrkmtz5m

Error:

dependency_failed: Not starting as dependency failed
Action:

generate_notebook_updating

Status:

Failed

Job identifier:

uuc3tn3ojzjf4mvv

Error:

dependency_failed: Not starting as dependency failed
Action:

join_ethnicity

Status:

Failed

Job identifier:

ofd2kvbymx7zmmkk

Error:

dependency_failed: Not starting as dependency failed
Action:

get_practice_count

Status:

Failed

Job identifier:

p76vg7hbc36pyidh

Error:

dependency_failed: Not starting as dependency failed
Action:

get_patient_count

Status:

Failed

Job identifier:

vfr236d4e2rac3qe

Error:

dependency_failed: Not starting as dependency failed

Pipeline

Show project.yaml

version: "3.0"

expectations:
  population_size: 1000

actions:
  
  generate_study_population_population:
    run: cohortextractor:latest generate_cohort --study-definition study_definition_population --index-date-range "2019-01-01 to 2023-06-01 by month" --output-dir=output --output-format=feather
    outputs:
      highly_sensitive:
        cohort: output/input_population*.feather

  join_ethnicity_population:
    run: >
      cohort-joiner:v0.0.46
        --lhs output/input_population_20*.feather
        --rhs output/input_ethnicity.feather
        --output-dir output/joined
    needs: [generate_study_population_population, generate_study_population_ethnicity]
    outputs:
      highly_sensitive:
        cohort: output/joined/input_population_20*.feather

  get_moved_count:
    run: python:latest python analysis/population_count.py
    needs: [join_ethnicity_population]
    outputs:
      moderately_sensitive:
        text: output/move*.json


  generate_study_population_1:
    run: cohortextractor:latest generate_cohort --study-definition study_definition --index-date-range "2019-01-01 to 2019-12-01 by month" --output-dir=output --output-format=feather
    outputs:
      highly_sensitive:
        cohort: output/input_*.feather

  generate_study_population_2:
    run: cohortextractor:latest generate_cohort --study-definition study_definition --index-date-range "2020-01-01 to 2020-12-01 by month" --output-dir=output --output-format=feather
    outputs:
      highly_sensitive:
        cohort: output/input*.feather

  generate_study_population_3:
    run: cohortextractor:latest generate_cohort --study-definition study_definition --index-date-range "2021-01-01 to 2021-06-01 by month" --output-dir=output --output-format=feather
    outputs:
      highly_sensitive:
        cohort: output/inpu*.feather

  generate_study_population_4:
    run: cohortextractor:latest generate_cohort --study-definition study_definition --index-date-range "2021-07-01 to 2021-12-01 by month" --output-dir=output --output-format=feather
    outputs:
      highly_sensitive:
        cohort: output/in*.feather

  generate_study_population_5:
    run: cohortextractor:latest generate_cohort --study-definition study_definition --index-date-range "2022-01-01 to 2023-06-01 by month" --output-dir=output --output-format=feather
    outputs:
      highly_sensitive:
        cohort: output/i*.feather


  generate_study_population_ethnicity:
    run: cohortextractor:latest generate_cohort --study-definition study_definition_ethnicity --output-dir=output --output-format=feather
    outputs:
      highly_sensitive:
        cohort: output/input_ethnicity.feather

  join_ethnicity:
    run: python:latest python analysis/join_ethnicity.py
    needs:
      [
        generate_study_population_1,
        generate_study_population_2,
        generate_study_population_3,
        generate_study_population_4,
        generate_study_population_5,
        generate_study_population_ethnicity,
      ]
    outputs:
      highly_sensitive:
        cohort: output/inp*.feather

  get_patient_count:
    run: python:latest python analysis/get_patients_counts.py
    needs: [join_ethnicity]
    outputs:
      moderately_sensitive:
        text: output/patient_count.json
      
  get_practice_count:
    run: python:latest python analysis/get_practice_count.py
    needs: [join_ethnicity]
    outputs:
      moderately_sensitive:
        text: output/practice_count.json

  generate_measures:
    run: cohortextractor:latest generate_measures --study-definition study_definition --output-dir=output
    needs: [join_ethnicity]
    outputs:
      moderately_sensitive:
        measure_csv: output/measure_*_rate.csv

  generate_measures_cleaned:
    run: python:latest python analysis/clean_measures.py
    needs: [generate_measures]
    outputs:
      moderately_sensitive:
        measure_csv: output/measure_cleaned_*.csv


  generate_notebook:
    run: jupyter:latest jupyter nbconvert /workspace/analysis/sentinel_measures.ipynb --execute --to html --template basic --output-dir=/workspace/output --ExecutePreprocessor.timeout=86400 --no-input
    needs:
      [
        generate_measures,
        generate_measures_cleaned,
        get_practice_count,
        get_patient_count,
      ]
    outputs:
      moderately_sensitive:
        notebook: output/sentinel_measures.html
        subplots: output/sentinel_measures_subplots.png
        code_tables: output/code_table_*.csv
        events_count: output/event_count.json

  generate_notebook_updating:
      run: jupyter:latest jupyter nbconvert /workspace/analysis/sentinel_measures_updating.ipynb --execute --to html --template basic --output-dir=/workspace/output --ExecutePreprocessor.timeout=86400 --no-input
      needs:
        [
          generate_measures,
          generate_measures_cleaned,
          get_practice_count,
          get_patient_count,
        ]
      outputs:
        moderately_sensitive:
          notebook: output/sentinel_measures_updating.html
          code_tables: output/code_table*.csv
          practices: output/num_practices_included*.csv

  run_tests:
    run: python:latest python -m pytest --junit-xml=output/pytest.xml --verbose
    outputs:
      moderately_sensitive:
        log: output/pytest.xml

  measures_ehrql:
    run: ehrql:v0 generate-measures analysis/dataset_definition.py --output output/measures.csv
    outputs:
      moderately_sensitive:
        measure_csv: output/measures.csv

  generate_notebook_updating_ehrql:
    run: jupyter:latest jupyter nbconvert /workspace/analysis/sentinel_measures_updating_ehrql.ipynb --execute --to html --template basic --output-dir=/workspace/output --ExecutePreprocessor.timeout=86400 --no-input
    needs:
      [measures_ehrql]
    outputs:
      moderately_sensitive:
        notebook: output/sentinel_measures_updating_ehrql.html

Timeline

Created: 2 years, 9 months ago 24 Jul 2023 17:44:58 UTC
Started: 2 years, 9 months ago 24 Jul 2023 17:43:20 UTC
Finished: 2 years, 9 months ago 29 Jul 2023 05:45:13 UTC
Runtime: 69:27:03

These timestamps are generated and stored using the UTC timezone on the TPP backend.

Job request

Status: Failed
Backend: TPP
Workspace: sro-measures-dashboard-updating
Requested by: Rose Higgins
Branch: main
Force run dependencies: Yes
Git commit hash: 7b54eeb
Requested actions: generate_study_population_population

join_ethnicity_population

get_moved_count

generate_study_population_1

generate_study_population_2

generate_study_population_3

generate_study_population_4

generate_study_population_5

generate_study_population_ethnicity

join_ethnicity

get_patient_count

get_practice_count

generate_measures

generate_measures_cleaned

generate_notebook

generate_notebook_updating

run_tests

Code comparison

Compare the code used in this job request