PATCH: A Plug-in Framework of Non-blocking Inference for Distributed Multimodal Systems

Conference: UbiComp 2023

PATCH is a plug-in framework for non-blocking inference in distributed multimodal systems under missing, delayed, or corrupted sensor streams. It introduces cross-modality feature imputation, lightweight feature pair ranking, and data alignment modules to preserve inference accuracy and low latency without retraining the original multimodal models.

Highlights

• Non-blocking multimodal inference under missing or delayed sensor data
• Plug-in framework compatible with early, intermediate, and late fusion models
• Cross-modality feature imputation with lightweight ranking and alignment modules
• Evaluated on nine multimodal models across autonomous driving, activity recognition, and event parsing tasks
• Up to 13% accuracy improvement with 73% lower training overhead than retraining