Skip to content

Job request: 24415

Organisation:
University of Bristol
Workspace:
post-covid-neurodegenerative-v1
ID:
5jbwgnhrih5gn4ws
View repo View project.yaml

This page shows the technical details of what happened when the authorised researcher Harry T Mason requested one or more actions to be run against real patient data in the project, within a secure environment.

By cross-referencing the list of jobs with the pipeline section below, you can infer what security level various outputs were written to. Researchers can never directly view outputs marked as highly_sensitive ; they can only request that code runs against them. Outputs marked as moderately_sensitive can be viewed by an approved researcher by logging into a highly secure environment. Only outputs marked as moderately_sensitive can be requested for release to the public, via a controlled output review service.

Jobs

Pipeline

Show project.yaml 
version: '3.0'

expectations:

  population_size: 5000

actions:

  ## # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # 
  ## DO NOT EDIT project.yaml DIRECTLY 
  ## This file is created by create_project_actions.R 
  ## Edit and run create_project_actions.R to update the project.yaml 
  ## # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # 
  ## Define study dates 

  study_dates:
    run: r:latest analysis/study_dates.R
    outputs:
      highly_sensitive:
        study_dates_json: output/study_dates.json

  ## Generate dates for all cohorts 

  generate_dates:
    run: ehrql:v1 generate-dataset analysis/dataset_definition/dataset_definition_dates.py
      --output output/dataset_definition/index_dates.csv.gz
    needs:
    - study_dates
    outputs:
      highly_sensitive:
        dataset: output/dataset_definition/index_dates.csv.gz

  ## Generate cohort - prevax 

  generate_cohort_prevax:
    run: ehrql:v1 generate-dataset analysis/dataset_definition/dataset_definition_prevax.py
      --output output/dataset_definition/input_prevax.csv.gz
    needs:
    - generate_dates
    outputs:
      highly_sensitive:
        cohort: output/dataset_definition/input_prevax.csv.gz

  ## Generate cohort - vax 

  generate_cohort_vax:
    run: ehrql:v1 generate-dataset analysis/dataset_definition/dataset_definition_vax.py
      --output output/dataset_definition/input_vax.csv.gz
    needs:
    - generate_dates
    outputs:
      highly_sensitive:
        cohort: output/dataset_definition/input_vax.csv.gz

  ## Generate cohort - unvax 

  generate_cohort_unvax:
    run: ehrql:v1 generate-dataset analysis/dataset_definition/dataset_definition_unvax.py
      --output output/dataset_definition/input_unvax.csv.gz
    needs:
    - generate_dates
    outputs:
      highly_sensitive:
        cohort: output/dataset_definition/input_unvax.csv.gz

  ## Clean data - prevax, with describe = TRUE 

  clean_data_prevax:
    run: r:latest analysis/dataset_clean/dataset_clean.R prevax TRUE
    needs:
    - study_dates
    - generate_cohort_prevax
    outputs:
      moderately_sensitive:
        describe_raw: output/describe/prevax_raw.txt
        describe_venn: output/describe/prevax_venn.txt
        describe_preprocessed: output/describe/prevax_preprocessed.txt
        flow: output/dataset_clean/flow_prevax.csv
        flow_midpoint6: output/dataset_clean/flow_prevax_midpoint6.csv
      highly_sensitive:
        venn: output/dataset_clean/venn_prevax.rds
        cohort_clean: output/dataset_clean/input_prevax_clean.rds

  ## Clean data - vax, with describe = TRUE 

  clean_data_vax:
    run: r:latest analysis/dataset_clean/dataset_clean.R vax TRUE
    needs:
    - study_dates
    - generate_cohort_vax
    outputs:
      moderately_sensitive:
        describe_raw: output/describe/vax_raw.txt
        describe_venn: output/describe/vax_venn.txt
        describe_preprocessed: output/describe/vax_preprocessed.txt
        flow: output/dataset_clean/flow_vax.csv
        flow_midpoint6: output/dataset_clean/flow_vax_midpoint6.csv
      highly_sensitive:
        venn: output/dataset_clean/venn_vax.rds
        cohort_clean: output/dataset_clean/input_vax_clean.rds

  ## Clean data - unvax, with describe = TRUE 

  clean_data_unvax:
    run: r:latest analysis/dataset_clean/dataset_clean.R unvax TRUE
    needs:
    - study_dates
    - generate_cohort_unvax
    outputs:
      moderately_sensitive:
        describe_raw: output/describe/unvax_raw.txt
        describe_venn: output/describe/unvax_venn.txt
        describe_preprocessed: output/describe/unvax_preprocessed.txt
        flow: output/dataset_clean/flow_unvax.csv
        flow_midpoint6: output/dataset_clean/flow_unvax_midpoint6.csv
      highly_sensitive:
        venn: output/dataset_clean/venn_unvax.rds
        cohort_clean: output/dataset_clean/input_unvax_clean.rds

Timeline

  • Created:

  • Started:

  • Finished:

  • Runtime: 08:21:46

These timestamps are generated and stored using the UTC timezone on the TPP backend.

Job information

Status
Succeeded
Backend
TPP
Workspace
post-covid-neurodegenerative-v1
Requested by
Harry T Mason
Branch
main
Force run dependencies
No
Git commit hash
dd8aa81
Requested actions
  • clean_data_prevax
  • clean_data_vax
  • clean_data_unvax

Code comparison

Compare the code used in this Job Request