Welcome to the flowMagic github repository!
flowMagic is an automated gating tool designed to automate the gating of bivariate flow cytometry (FCM) data. The flowMagic algorithm is the first algorithm trained on both expert-generated data and a large dataset of gated bivariate FCM data annotated by the players of EVE Online, the online role-playing game developed by the CCP Games company.
Flow cytometry (FCM) is a technology widely used in immunology and cell biology to enable the precise identification and quantification of heterogeneous cell populations across diverse applications. The prevailing analysis approach, termed gating, relies on expert manual analysis of hierarchical bivariate plots. This is a subjective and time-consuming approach that generates inconsistent results. In order to generate a large high quality training dataset in a relatively short amount of time, we recruited the players of EVE Online to gate millions of bivariate FCM plots within the mini-game Project Discovery. The flowMagic algorithm, trained on this large diverse dataset, represents a significant advancement in ML-based analysis of FCM data, combining citizen science and machine learning to develop more robust, generalizable automated FCM analysis tools.
flowMagic framework includes several key functionalities:
-
Rich visualization tools for intuitive exploration of bivariate FCM data. The flowMagic framework works seamlessly with the flowCore/flowWorkspace ecosystem.
-
Interactive manual gating for user-driven analysis.
-
Automated gating with machine learning, offering two models:
- Template model: trained on expert-generated data provided by the user based on the panel to analyze.
- Generalized model: trained on the data generated by the videogamers (no template data required).
-
Improved navigation utilities for working with flowCore and flowWorkspace objects.
All of these functionalities are actively maintained and continuously updated to improve performance, usability and compatibility with the broader FCM analysis ecosystem.
Download Full Introduction Video
The full technical documentation for the flowMagic R package can be found inside the pk_manual directory above. Click on the red pdf badge (flowMagic vignette) to download and visualize the flowMagic introduction (vignette). Click on the red pdf badge named flowMagic manual to download a guide to all flowMagic functions and arguments.
The flowMagic package can be installed within R using the devtools library:
library(devtools)
install_github("semontante/flowMagic")Alternatively, the user can download the flowMagic package and install it from their local directory:
install.packages("path/to/flowMagic.tar.gz",repos=NULL,type="source")The script below can be used to test the correct installation of the flowMagic package.
# load libraries
library(sp)
library(stringr)
library(ggplot2)
library(parallel)
library(doParallel)
library(randomForest)
library(caret)
library(concaveman)
library(sm)
library(pracma)
library(sf)
library(stats)
library(grDevices)
library(flowMagic)
#------------ using template model with 1 template
# get path to directory with files to analyze
path_dir<-system.file("extdata/csv_files",package = "flowMagic")
# import data with labels that we use as template data.
list_data_ref<-import_reference_csv(path_results = path_dir,n_cores = 1)
# import data without labels
list_test_data<-import_test_set_csv(path_data = path_dir,n_cores = 1)
# Note that it is possible to provide also directly the paths to each file.
# See functions manual for additional details.
# data preprocessing to generate the template model using first file as template
# we select first element of the imported list of dataframes
ref_train<-get_train_data(paths_file = list_data_ref[1],n_cores = 1)
# generate template model using out-of-the-bag validation
ref_model_info<-magicTrain(df_train = ref_train,n_cores = 1,train_model = "rf")
# perform automated gating (gates boundaries prediction step)
list_dfs_pred<-magicPred_all(list_test_data = list_test_data,magic_model = NULL,ref_data_train = ref_train,
ref_model_info = ref_model_info,n_cores = 8)
# Note that providing the training set is optional (ref_data_train = ref_train is optional).
# Providing the training set allows the user to calculate the target-template distance for each plot to analyze.
# list_dfs_pred contains a list of dataframes for each plot analyzed. In other words, it is a nested list (e.g.,
# downsampled dataset and original dataset with predicted labels for each plot).
# See the functions manual for the full list of dataframes returned.
# visualize gated data
df_temp<-list_dfs_pred[[1]]$df_test_original # dataframe of first gated plot
magicPlot(df = df_temp,type = "ML",size_points = 1)
magicPlot(df = df_temp,type = "dens",size_points = 1)This R package is licensed under the Apache License 2.0. See the LICENSE and NOTICE files for more information.
