Skip to content

Job request: 27154

Organisation:
University of Bristol
Workspace:
depill-ckd-1
ID:
u3zfzttne2gsmdb5
View repo View project.yaml

This page shows the technical details of what happened when the authorised researcher Matthew Letts requested one or more actions to be run against real patient data within a secure environment.

By cross-referencing the list of jobs with the pipeline section below, you can infer what security level the outputs were written to.

The output security levels are:

  • highly_sensitive
    • Researchers can never directly view these outputs
    • Researchers can only request code is run against them
  • moderately_sensitive
    • Can be viewed by an approved researcher by logging into a highly secure environment
    • These are the only outputs that can be requested for public release via a controlled output review service.

Jobs

  • Action:
    dataset_cleaning_inex
    Status:
    Succeeded
    Job identifier:
    t554wyd3gs2tcher
  • Action:
    dataset_process_baseline_meds
    Status:
    Pending
    Job identifier:
    owf7l3pe7v3xcrx4
    Status message:
    waiting_on_dependencies: Waiting on dependencies
  • Action:
    analyse_baseline_meds
    Status:
    Pending
    Job identifier:
    ikjpjbg27ms44un6
    Status message:
    waiting_on_dependencies: Waiting on dependencies
  • Action:
    generate_dataset_baseline_meds
    Status:
    Pending
    Job identifier:
    om6sgsun47a2zq7m
    Status message:
    waiting_on_db_workers: Waiting on available database workers

Pipeline

Show project.yaml 
version: '5.0'

actions:

  study_dates:
    run: r:v3 analysis/config/config.r
    outputs:
      highly_sensitive:
        study_dates_json: output/study_dates.json


  generate_dataset_inex:
    run: >
        ehrql:v1 generate-dataset analysis/dataset_definition/dataset_definition_inex.py
        --dummy-tables dummy_tables/
        --output output/dataset_inex.arrow
      # --test-data-file analysis/dataset_definition/test_dataset_definition_inex.py
    needs: [study_dates]
    outputs:
      highly_sensitive:
        dataset_inex: output/dataset_inex.arrow


  dataset_cleaning_inex:
    run: r:v3 analysis/dataset_processing/dataset_cleaning_inex.r
    needs: [generate_dataset_inex]
    outputs:
      highly_sensitive:
        dataset_inex_cleaned: output/data/dataset_inex_cleaned.arrow
      moderately_sensitive:
        cleaning_inex_data_flow: output/data_descriptions/cleaning_inex/data_flow.csv
        cleaning_inex_data_descriptions: output/data_descriptions/cleaning_inex/*.txt
        cleaning_inex_med_count_summary: output/data_descriptions/cleaning_inex/med_count_summary.csv
        cleaning_inex_ckd_sensitivity_counts: output/data_descriptions/cleaning_inex/ckd_sensitivity_counts.csv

  
  generate_dataset_baseline_covariates_general:
    run: >
      ehrql:v1 generate-dataset analysis/dataset_definition/dataset_definition_baseline_covariates_general.py
      --dummy-tables dummy_tables/
      --output output/dataset_baseline_covariates_general.arrow
    #  --test-data-file analysis/dataset_definition/test_dataset_definition_baseline_covariates_general.py
    needs: [study_dates, dataset_cleaning_inex]
    outputs:
      highly_sensitive:
        dataset_baseline_covariates_general: output/dataset_baseline_covariates_general.arrow


  # generate_dataset_baseline_covariates_med_specific_ppi:
  #   run: >
  #     ehrql:v1 generate-dataset analysis/dataset_definition/dataset_definition_baseline_covariates_med_specific.py
  #     --dummy-tables dummy_tables/
  #     --output output/dataset_baseline_covariates_med_specific_ppi.arrow
  #     -- --medicine ppi
  #   needs: [study_dates, dataset_cleaning_inex]
  #   outputs:
  #     highly_sensitive:
  #       dataset_baseline_covariates_med_specific_ppi: output/dataset_baseline_covariates_med_specific_ppi.arrow


  # generate_dataset_baseline_covariates_med_specific_statin:
  #   run: >
  #     ehrql:v1 generate-dataset analysis/dataset_definition/dataset_definition_baseline_covariates_med_specific.py
  #     --dummy-tables dummy_tables/
  #     --output output/dataset_baseline_covariates_med_specific_statin.arrow
  #     -- --medicine statin
  #   needs: [study_dates, dataset_cleaning_inex]
  #   outputs:
  #     highly_sensitive:
  #       dataset_baseline_covariates_med_specific_ppi: output/dataset_baseline_covariates_med_specific_statin.arrow


  generate_dataset_baseline_meds:
    run: >
      ehrql:v1 generate-dataset analysis/dataset_definition/dataset_definition_baseline_meds.py
      --dummy-tables dummy_tables/
      --output output/dataset_baseline_meds.arrow
    # --test-data-file analysis/dataset_definition/test_dataset_definition_baseline_meds.py
    needs: [study_dates, dataset_cleaning_inex]
    outputs:
      highly_sensitive:
        dataset_baseline_meds: output/dataset_baseline_meds.arrow


  dataset_process_baseline_meds:
    run: r:v3 analysis/dataset_processing/dataset_process_baseline_meds.r
    needs: [generate_dataset_baseline_meds, study_dates]
    outputs:
      highly_sensitive:
        dataset_baseline_meds_processed: output/data/dataset_baseline_meds_processed.arrow
      moderately_sensitive:
        process_baseline_meds_data_flow: output/data_descriptions/process_baseline_meds/data_flow.csv
        process_baseline_meds_dmd_not_in_lookup: output/data_descriptions/process_baseline_meds/dmd_not_in_lookup.csv
        process_baseline_meds_dmd_unmapped_to_bnf: output/data_descriptions/process_baseline_meds/dmd_unmapped_to_bnf.csv
        process_baseline_meds_data_descriptions: output/data_descriptions/process_baseline_meds/*.txt
        process_baseline_meds_route_classification_patient_summary: output/data_descriptions/process_baseline_meds/route_classification_patient_summary.csv


  analyse_baseline_meds:
    run: r:v3 analysis/dataset_analysis/analyse_baseline_meds.r
    needs: [dataset_process_baseline_meds, study_dates]
    outputs:
      highly_sensitive:
        dataset_baseline_meds_analysed: output/data/dataset_baseline_meds_analysed.arrow
      moderately_sensitive:
        analyse_baseline_meds_data_flow: output/data_descriptions/analyse_baseline_meds/data_flow.csv
        counts_days_before_index: output/data_descriptions/analyse_baseline_meds/counts_days_before_index.csv
        chronic_meds_summary_sens_analysis: output/data_descriptions/analyse_baseline_meds/chronic_meds_summary.csv
        chronic_meds_frequency_sens_analysis: output/data_descriptions/analyse_baseline_meds/chronic_meds_frequency.csv
        analyse_baseline_meds_data_descriptions: output/data_descriptions/analyse_baseline_meds/*.txt
        meds_prevalence_substance: output/data_descriptions/analyse_baseline_meds/meds_prevalence_substance.csv
        meds_prevalence_subparagraph: output/data_descriptions/analyse_baseline_meds/meds_prevalence_subparagraph.csv
        meds_prevalence_paragraph: output/data_descriptions/analyse_baseline_meds/meds_prevalence_paragraph.csv
        bnf_chapters_frequency: output/data_descriptions/analyse_baseline_meds/bnf_chapters_frequency.csv


  table_1:
    run: r:v3 analysis/dataset_analysis/table_1.r
    needs: [dataset_cleaning_inex]
    outputs:
      moderately_sensitive:
        table_1: output/tables/table_1.csv


  generate_dataset_discont_meds_baseline_statins:
    run: >
      ehrql:v1 generate-dataset analysis/dataset_definition/dataset_definition_discont_meds_baseline.py
      --dummy-tables dummy_tables/
      --output output/dataset_discont_meds_baseline_statins.arrow
      --
      --medication statins
    needs: [study_dates, dataset_cleaning_inex]
    outputs:
      highly_sensitive:
        dataset_discont_meds_baseline_statins: output/dataset_discont_meds_baseline_statins.arrow


  generate_dataset_discont_meds_baseline_ppis:
    run: >
      ehrql:v1 generate-dataset analysis/dataset_definition/dataset_definition_discont_meds_baseline.py
      --dummy-tables dummy_tables/
      --output output/dataset_discont_meds_baseline_ppis.arrow
      --
      --medication ppis
    needs: [study_dates, dataset_cleaning_inex]
    outputs:
      highly_sensitive:
        dataset_discont_meds_baseline_ppis: output/dataset_discont_meds_baseline_ppis.arrow
  # add another parameterised action to pull data up to index date for each moi


  dataset_process_moi_at_baseline:
    run: r:v3 analysis/discontinuation/dataset_processing/dataset_process_moi_at_baseline.r
    needs: [generate_dataset_discont_meds_baseline_statins, generate_dataset_discont_meds_baseline_ppis, study_dates]
    outputs:
      highly_sensitive:
        dataset_statins_at_baseline_processed: output/data/dataset_statins_at_baseline_processed.arrow
        dataset_statins_at_baseline_data_descriptions: output/data_descriptions/process_moi_at_baseline_statins/*.txt
        dataset_ppis_at_baseline_processed: output/data/dataset_ppis_at_baseline_processed.arrow
        dataset_ppis_at_baseline_data_descriptions: output/data_descriptions/process_moi_at_baseline_ppis/*.txt
      moderately_sensitive:
        statins_at_baseline_data_flow: output/data_descriptions/process_moi_at_baseline_statins/data_flow.csv
        ppis_at_baseline_data_flow: output/data_descriptions/process_moi_at_baseline_ppis/data_flow.csv
        # add more needs/outputs for each medication of interest


  generate_dataset_discont_meds_follow_up_statins:
    run: >
      ehrql:v1 generate-dataset analysis/dataset_definition/dataset_definition_discont_meds_follow_up.py
      --dummy-tables dummy_tables/
      --output output/dataset_discont_meds_follow_up_statins.arrow
      --
      --medication statins
    needs: [study_dates, dataset_process_moi_at_baseline]
    outputs:
      highly_sensitive:
        dataset_discont_meds_follow_up_statins: output/dataset_discont_meds_follow_up_statins.arrow


  generate_dataset_discont_meds_follow_up_ppis:
    run: >
      ehrql:v1 generate-dataset analysis/dataset_definition/dataset_definition_discont_meds_follow_up.py
      --dummy-tables dummy_tables/
      --output output/dataset_discont_meds_follow_up_ppis.arrow
      --
      --medication ppis
    needs: [study_dates, dataset_process_moi_at_baseline]
    outputs:
      highly_sensitive:
        dataset_discont_meds_follow_up_ppis: output/dataset_discont_meds_follow_up_ppis.arrow
    # add another parameterised action to pull followup data for each moi

Job statistics

Status Count Percentage
Pending 3 75%
Running 0 0%
Succeeded 1 25%
Failed 0 0%

1 / 4 (25%) complete

Timeline

  • Created:

  • Started:

  • Runtime: 00:22:56

These timestamps are generated and stored using the UTC timezone on the TPP backend.

Job request

Status
Running
Backend
TPP
Workspace
depill-ckd-1
Requested by
Matthew Letts
Branch
main
Force run dependencies
No
Git commit hash
dfcedc7
Requested actions
  • dataset_cleaning_inex
  • generate_dataset_baseline_meds
  • dataset_process_baseline_meds
  • analyse_baseline_meds

Code comparison

Compare the code used in this job request