Healthcare data pipelines can't afford to fail. When patient care and regulatory compliance depend on data availability, resilience isn't optional - it's fundamental. We share practical patterns and hard-won lessons from building a fault-tolerant ETL system that processes critical healthcare data from multiple sources while maintaining 99.9% uptime.
Testing healthcare ETL pipelines presents a fundamental dilemma: how do you ensure your data transformations work correctly without exposing sensitive patient information? With GDPR fines reaching €20 million and patient trust at stake, we developed a synthetic data generation approach that enables comprehensive testing while maintaining complete isolation from real patient data.
Healthcare data processing presents unique challenges - multiple data sources, stringent compliance requirements, and the critical nature of accuracy. We recently tackled these challenges by building a modern ETL pipeline using Dagster's asset-centric approach, moving beyond traditional task-based orchestration to create a more maintainable and observable data platform.
This is the first prototype of a 3D dataviz component that allows dataset (rows and columns) to be displayed and explored with seamless visual transitions through various presentations defined by the data field types.
We extend our 3D (VR) dataviz component to visualize an election datasets, showing a geographic hex-map for the UK constituencies and a radial chart showing the party sizes.
We used our 3D (VR) dataviz component to visualize the UK 2024 election, comparing it to the 2019 election, which shows a dramatic change of fortune for the two main UK parties.
We used D3 and Svelte (with Layercake) to build an in-house project management app for Queen Mary's College London (QMUL). Using a pseudo-physics force-directed network allows one to find correlations in the project dataset.
This chart transitions from a conventional multi-line chart to a stacked area chart, using Svelte transitions. LayerCake and D3 add ballast to the foundations. The y axis can be switched from fixed to dynamic.
A client wanted a recreation of the rather superb binary tree demo by R2D3. Without appropriating the specific aesthetics but building a configurable component that could be used to demonstrate binary trees in action and teach machine-learning.
We used Svelte (with Layercake) to build a little scatterplot component with selectable axes and toggled log-plots. This makes for an easy Gapminder-alike demo with the classic income/population/life-expectancy dataset for countries.
We used Svelte (with Layercake) to build a minimal app as a prelimary for an in-house tool for analyzing large pollen-count datasets. With customized dataviz the principal features and correlations of the dataset can be brought into sharp focus.
In order to present their data on discrimination against Asian-Americans in the United States, Dataface required some bespoke interactive charts, based on some attractive inhouse designs. They wanted the charts done with Svelte and suggested using the LayerCake library, which provides a clever Svelte based scaffold for D3 etc. dataviz.
This visualisation uses a force-directed layout, courtesy of D3, to view approximately 1200 classic Jazz standards by key, date and composer. Some interesting patterns emerge.
A data visualisation for the UK government's Verify project, allowing exploration of Verify''s demographic coverage using a dataset provided by the Office of National Statistics
