Job request: 21845

Organisation:: Bennett Institute
Workspace:: opioids-covid-research
ID:: txxnabt2dqxn2cx5

This page shows the technical details of what happened when the authorised researcher Andrea Schaffer requested one or more actions to be run against real patient data within a secure environment.

By cross-referencing the list of jobs with the pipeline section below, you can infer what security level the outputs were written to.

The output security levels are:

highly_sensitive
- Researchers can never directly view these outputs
- Researchers can only request code is run against them
moderately_sensitive
- Can be viewed by an approved researcher by logging into a highly secure environment
- These are the only outputs that can be requested for public release via a controlled output review service.

Jobs

Action:

generate_dataset_table

Status:

Status: Succeeded

Job identifier:

duixrhzx2kpy3ue3
Action:

measures_carehome

Status:

Status: Succeeded

Job identifier:

l6gr74wc533xmayo
Action:

measures_demo_new

Status:

Status: Succeeded

Job identifier:

ev57irdjelbcu7wa
Action:

measures_demo_prev

Status:

Status: Succeeded

Job identifier:

jk4vz37txwxvytpx
Action:

table

Status:

Status: Succeeded

Job identifier:

ryed5toy3f4o24cl
Action:

measures_overall

Status:

Status: Succeeded

Job identifier:

b4gyhicur3kpowsd
Action:

process_ts_carehome

Status:

Status: Succeeded

Job identifier:

3cst6nulyvzlttuo
Action:

process_ts_demo

Status:

Status: Succeeded

Job identifier:

dnwtbaxervul4fr5
Action:

process_ts_overall

Status:

Status: Succeeded

Job identifier:

sobukob3n3aoq7ug
Action:

measures_test

Status:

Status: Failed

Job identifier:

foygao7o4rnjdtko

Error:

cancelled_by_user: Cancelled by user
Action:

figures_ts

Status:

Status: Failed

Job identifier:

lwbyatzxsbc32ma3

Error:

dependency_failed: Not starting as dependency failed
Action:

measures_type

Status:

Status: Failed

Job identifier:

fhgv4enyrz64infp

Error:

cancelled_by_user: Cancelled by user
Action:

process_ts_type

Status:

Status: Failed

Job identifier:

3zktpo332p4fer54

Error:

dependency_failed: Not starting as dependency failed

Pipeline

Show project.yaml

######################################
# This script defines the project pipeline - 
# it specifies the execution orders for all the code in this
# repo using a series of actions.
######################################


expectations:

  population_size: 10000

version: '3.0'

actions:

  generate_dataset_table:
    run: ehrql:v1 generate-dataset analysis/define_dataset_table.py 
      --output output/data/dataset_table.csv.gz
    outputs:
      highly_sensitive:
        cohort: output/data/dataset_table.csv.gz  

  # Measures - prevalent and new prescribing - overall
  measures_overall:
    run: ehrql:v1 generate-measures analysis/measures_overall.py 
      --output output/measures/measures_overall.csv
      --
      --start-date "2018-01-01"
      --intervals 54
    outputs:
      moderately_sensitive:
        measure_csv: output/measures/measures_overall.csv

  # Measures - prevalent prescribing - by demographic categories
  measures_demo_prev:
    run: ehrql:v1 generate-measures analysis/measures_demo_prev.py 
      --output output/measures/measures_demo_prev.csv
      --
      --start-date "2018-01-01"
      --intervals 54
    outputs:
      moderately_sensitive:
        measure_csv: output/measures/measures_demo_prev.csv
  
  # Measures - new prescribing - by demographic categories
  measures_demo_new:
    run: ehrql:v1 generate-measures analysis/measures_demo_new.py 
      --output output/measures/measures_demo_new.csv
      --
      --start-date "2018-01-01"
      --intervals 54
    outputs:
      moderately_sensitive:
        measure_csv: output/measures/measures_demo_new.csv

  # Measures - prevalent prescribing - by opioid type
  measures_type:
    run: ehrql:v1 generate-measures analysis/measures_type.py 
      --output output/measures/measures_type.csv
      --
      --start-date "2018-01-01"
      --intervals 54
    outputs:
      moderately_sensitive:
        measure_csv: output/measures/measures_type.csv 

  # Measures - prevalent and new prescribing - in people in care home
  measures_carehome:
    run: ehrql:v1 generate-measures analysis/measures_carehome.py 
      --output output/measures/measures_carehome.csv
      --
      --start-date "2018-01-01"
      --intervals 54
    outputs:
      moderately_sensitive:
        measure_csv: output/measures/measures_carehome.csv

  # Measures - prevalent prescribing - checking overall counts
  measures_test:
    run: ehrql:v1 generate-measures analysis/measures_test.py 
      --output output/measures/measures_test.csv
      --
      --start-date "2019-07-01"
      --intervals 36
    outputs:
      moderately_sensitive:
        measure_csv: output/measures/measures_test.csv 
        
  ## Process time series data - overall prescribing 
  process_ts_overall:
   run: r:latest analysis/process/process_ts_overall.R
   needs: [measures_overall]
   outputs:
      moderately_sensitive:
        timeseries_csv: output/timeseries/ts_overall*.csv
 
  ## Process time series data - prescribing by demographics
  process_ts_demo:
   run: r:latest analysis/process/process_ts_demo.R
   needs: [measures_demo_prev, measures_demo_new]
   outputs:
      moderately_sensitive:
        timeseries_csv: output/timeseries/ts_demo*.csv
  
  ## Process time series data - prescribing by admin route
  process_ts_type:
   run: r:latest analysis/process/process_ts_type.R
   needs: [measures_type]
   outputs:
      moderately_sensitive:
        timeseries_csv: output/timeseries/ts_type*.csv
  
  ## Process time series data - prescribing to people in carehome
  process_ts_carehome:
   run: r:latest analysis/process/process_ts_carehome.R
   needs: [measures_carehome]
   outputs:
      moderately_sensitive:
        timeseries_csv: output/timeseries/ts_carehome*.csv

  ## Check time series
  figures_ts:
    run: r:latest analysis/descriptive/ts_figures.R
    needs: [process_ts_overall, process_ts_type, process_ts_carehome, process_ts_demo]
    outputs:
      moderately_sensitive:
        plots: output/descriptive/ts_plot*.png

  ## Results table 
  table:
    run: r:latest analysis/descriptive/table_stand.R
    needs: [generate_dataset_table]
    outputs:
      moderately_sensitive:
        table: output/tables/table_*.csv



  # OLD COHORTEXTRACTOR CODE

  # ## Cohort data
  # generate_study_population_1:
  #   run: cohortextractor:latest generate_cohort
  #     --study-definition study_definition
  #     --index-date-range "2018-01-01 to 2018-12-01 by month" 
  #     --output-dir=output 
  #     --output-format=csv
  #   outputs:
  #     highly_sensitive:
  #       cohort: output/input_*.csv

  # generate_study_population_2:
  #   run: cohortextractor:latest generate_cohort 
  #     --study-definition study_definition
  #     --index-date-range "2019-01-01 to 2019-12-01 by month" 
  #     --output-dir=output 
  #     --output-format=csv
  #   outputs:
  #     highly_sensitive:
  #       cohort: output/input*.csv

  # generate_study_population_3:
  #   run: cohortextractor:latest generate_cohort 
  #     --study-definition study_definition
  #     --index-date-range "2020-01-01 to 2020-12-01 by month" 
  #     --output-dir=output 
  #     --output-format=csv
  #   outputs:
  #     highly_sensitive:
  #       cohort: output/inpu*.csv

  # generate_study_population_4:
  #   run: cohortextractor:latest generate_cohort 
  #     --study-definition study_definition
  #     --index-date-range "2021-01-01 to 2021-12-01 by month" 
  #     --output-dir=output 
  #     --output-format=csv
  #   outputs:
  #     highly_sensitive:
  #       cohort: output/inp*.csv
  
  # generate_study_population_5:
  #   run: cohortextractor:latest generate_cohort 
  #     --study-definition study_definition
  #     --index-date-range "2022-01-01 to 2022-03-01 by month" 
  #     --output-dir=output 
  #     --output-format=csv
  #   outputs:
  #     highly_sensitive:
  #       cohort: output/in*.csv

  # ## Ethnicity      
  # generate_ethnicity_cohort:
  #   run: >
  #     cohortextractor:latest generate_cohort
  #       --study-definition study_definition_ethnicity
  #   outputs:
  #     highly_sensitive:
  #       cohort: output/input_ethnicity.csv


  # # Data processing ----
  
  # ## Add ethnicity
  # join_cohorts:
  #   run: >
  #     cohort-joiner:v0.0.48
  #       --lhs output/input_*.csv
  #       --rhs output/input_ethnicity.csv
  #       --output-dir output/data
  #   needs: [generate_study_population_1,  generate_study_population_2, 
  #     generate_study_population_5, generate_study_population_3, 
  #     generate_study_population_4, generate_ethnicity_cohort]
  #   outputs:
  #     highly_sensitive:
  #       cohort: output/data/input_*.csv 


  # ## Generate measures - full population
  # generate_measures:
  #   run: >
  #     cohortextractor:latest generate_measures 
  #       --study-definition study_definition
  #       --output-dir output/data
  #   needs: [join_cohorts]
  #   outputs:
  #     moderately_sensitive:
  #       measure_csv: output/data/measure_*.csv

Timeline

Created: 1 year, 10 months ago 27 Jan 2024 08:41:12 UTC
Started: 1 year, 10 months ago 27 Jan 2024 08:41:47 UTC
Finished: 1 year, 10 months ago 29 Jan 2024 10:42:32 UTC
Runtime: 121:13:56

These timestamps are generated and stored using the UTC timezone on the TPP backend.

Job request

Status: Failed
Backend: TPP
Workspace: opioids-covid-research
Requested by: Andrea Schaffer
Branch: main
Force run dependencies: No
Git commit hash: 52d5728
Requested actions: generate_dataset_table

measures_overall

measures_demo_prev

measures_demo_new

measures_type

measures_carehome

measures_test

process_ts_overall

process_ts_demo

process_ts_type

process_ts_carehome

figures_ts

table

Code comparison

Compare the code used in this job request