Projects | PEILab

A Unified TinyML System for Multi-modal Edge Intelligence and Real-time Visual Perception.

Wed, 20 Apr 2022 00:00:00 +0000

1. Introduction

Modern machine learning (ML) applications are often deployed in the cloud environment to exploit the computational power of clusters. However, this in-cloud computing scheme cannot satisfy the demands of emerging edge intelligence scenarios, including providing personalized models, protecting user privacy, adapting to real-time tasks and saving resource costs. To conquer the limitations of conventional in-cloud computing, it comes the rise of on-device learning, which handles the end-to-end ML procedure mainly on user devices, and restricts unnecessary involvement of the cloud. Despite the promising advantages of on-device learning, implementing a high-performance on-device learning system still faces many severe challenges, such as insufficient user training data, backward propagation blocking and limited peak processing speed.

Illustration: Conventional ML applications rely on the in-cloud learning paradigm, incurring essential drawbacks. Upgrading to the TinyML paradigm can effectively address these issues.

2. Architecture Overview

Observing the substantial improvement space in the implementation and acceleration of on-device learning systems, our group devote to designing high-performance TinyML architectures and relevant optimization algorithms, especially for embedded devices and microprocessors. Our research focuses on the software and hardware synergy of on-device learning techniques, covering the scope of model-level neural network design, algorithm-level training optimization and hardware-level instruction acceleration. Here, we present the architecture overview of our system design.

Architecture Overview

Illustration: an efficient TinyML system require a holistic design of the entire hierarchy, which can be resolved as five key research opportunities.

3. Research Opportunities

Here are five key research opportunities to implement our system. Please check the sub-folders for details.

4. Achievements

The on-device learning techniques can be employed in many emerging TinyML scenarios, where the system performance is often bounded by the limited hardware resources. Currently, our group has achieved breakthroughs in improving the computational capacity and designing domain-specific AI chips for task acceleration. These chips can be designed from the perspectives of model compression, few-shot learning, quantization-ware training, memory management and low-level instructions. We pursue the vision that helps researchers and developers optimize AI deployment without tedious code modifications. Some research demos have been open-source on Github, please visit at:

[1] Octo: INT8 Training with Loss-aware Compensation and Backward Quantization for Tiny On-device Learning, In Proc. of USENIX Annual Technical Conference (ATC), 2021 (CCF-A). [2] On-device Learning Systems for Edge Intelligence: A Software and Hardware Synergy Perspective, IEEE Internet of Things Journal, 2020 (JCR-Q1). [3] Petrel: Heterogeneity-aware Distributed Deep Learning via Hybrid Synchronization, IEEE Transactions on Parallel and Distributed Systems (TPDS), 2020 (CCF-A). [4] Dual-view Attention Networks for Single Image Super-Resolution, In Proc. of the ACM International Conference on Multimedia (MM), 2020 (CCF-A).

6. Cooperators

Our group have established close cooperation with industrial communities, including Microsoft Research Asia, Alibaba DAMO Academy, Huawei Cloud, etc.

7. PhD/intern Applications:

We are looking for students and partners who are interested in:

(1) On-device/TinyML Systems (for Edge Intelligence)
(2) Distributed Machine Learning Systems (for Data center)
(3) Modern AI/ML frameworks: e.g., NVIDIA NCCL, CUDA, TensorRT, Apple CoreML, PyTorch, TensorFlow, Keras, BytePS, Gym, etc.
(4) Domain-specific hardware optimization and implementation, e.g., NVIDIA Jetson, FPGA, Microprocessors, AI Chips, etc.
(5) Coding contribution to our GitHub repositories.

Adaptive Quantization-aware Training and Model Compression.

Mon, 18 Apr 2022 00:00:00 +0000

Research Opportunity 1: Adaptive Quantization-aware Training and Model Compression

Illustration: On-device learning is an emerging technique to pave the last mile of enabling edge intelligence, which eliminates the limitations of conventional in-cloud computing where dozens of computational capacities and memories are needed. A high- performance on-device learning system requires breaking the constraints of limited resources and alleviating computational overhead. Our preliminary work shows that employing the 8-bit fixed-point (INT8) quantization in both forward and back- ward passes over a deep model is a promising way to enable tiny on-device learning in practice. The key to an efficient quantization-aware training (QAT) method is to exploit the hardware- level enabled acceleration while preserving the training quality in each layer. However, off-the-shelf quantization methods cannot handle the on-device learning paradigm of fixed-point processing. To overcome these challenges, we propose to design an adaptive QAT algorithm, which jointly optimizes the computation of forward and backward passes. Besides, we need to build efficient network components to automatically counteract the quantization error of tensor arithmetic. We intend to implement our methods in Octo, a lightweight cross-platform system for tiny on-device learning, and keep improving its performance to support more realistic applications.

A Unified Contrastive Representation Learner for Cross-modal Federated Learning Systems.

Sat, 16 Apr 2022 00:00:00 +0000

Illustration: Contrastive representation learners have achieved great advantages for modern visual tasks. Existing methods (e.g., CLIP, visialGPT, VideoCLIP, and UniFormer) are resource-expensive, thus are not suitable for the realistic scenarios of deploying federated learning applications. Meanwhile, the single data modality of conventional FL systems significantly limits the scalability and applicability. Building an economical and efficient representation learner is the key issue to implement downstream tasks. This requires us to design a new cross-modal federated learning framework, which tackles the multimodality fusion of latent features and provides higher performance over the single-modal paradigms.

Progressive Network Sparsification and Latent Feature Compression for Scalable Collaborative Learning.

Fri, 15 Apr 2022 00:00:00 +0000

Research Opportunity 4: Progressive Network Sparsification and Latent Feature Compression for Scalable Collaborative Learning

Illustration: In the edge intelligence environment, new data is continuously generated on user devices that cannot be aggregated at once due to privacy and energy concerns. These issues require us to develop new insights into traffic saving to build a communication-efficient collaborative learning paradigm. Unlike previous methods aiming at improving bandwidth utilization or using an unstructured pixel-wise compression, we jointly capture the channel and spatial-level feature redundancy, and conduct a hierarchical compression in these two levels to achieve a much higher traffic reduction ratio. Specifically, we need to design a more efficient feature compression method to leverage the pixel similarity, and reorganize the features into groups based on channel significance to prune the network. Meanwhile, we intend to calibrate the gradients of compressed features with a comprehensive theoretical analysis of the convergence rate. Such a co-design can provide a significant traffic reduction over existing methods while not sacrificing much model accuracy, achieving good training flexibility and communicational efficiency. We believe this work can contribute to the further development of edge intelligence applications.

Masked Autoencoders for Occlusion-aware Visual Learners

Thu, 14 Apr 2022 00:00:00 +0000

Research Opportunity 5: Masked Autoencoders for Occlusion-aware Visual Learners

Illustration: Recent years have witnessed learning-based video perception algorithms getting popular in more scenarios with occlusions, where invisible areas for perception objects significantly affect accuracy. Existing methods mainly use convolutional neural networks as the backbone and get limited local features to recover the occluded part. Such an anti-occlusion pipeline often suffers from the challenges of self-occlusion scenery, where similar parts of occluders and occludes are ambiguous. In this case, we need to design a masked visual autoencoder for image processing and video streaming, which recovers occluded regions by extracting deep spatial information at a higher semantic level. This autoencoder can get better details inferred from global self-attention and thus improves accuracy. The gist is to train the autoencoder to extract key-point information from the key patches that are manually masked in a self-supervised manner to simulate the occlusion in video streaming. To choose the patches that should be masked, we design a high-capacity learnable gate that can extract contrastive representation, i.e., distinguish important feature regions and background regions, to generate a binary mask by randomly choosing a part of feature patches. We also propose an end-to-end pipeline for training and inference, which can effectively reduce the dependency of annotated occluded datasets and can be further applied to other visual tasks. This pipeline can obtain a great computation saving with much fewer annotated datasets, and hold a higher runtime performance over the SOTA ViT methods.

Flexible Patch Skip for Real-time Visual Perception.

Tue, 05 Apr 2022 00:00:00 +0000

Illustration: utilizing the temporal redundancy in video streams to construct efficient on-device video perception systems is a potential approach. We isolate the computation-saving challenge from video perception tasks and offer a task-independent acceleration approach that may be applied across a variety of runtime contexts. By separating acceleration and tasks, we plan to build novel quality-determining criteria for system design and provide an autonomous computation skipping approach to enable different video perception settings. We want to use a learnable gate in each convolution layer to decide which patches may be safely omitted without affecting model accuracy. The gate is optimized by a rigorous self-supervising approach that learns high-level semantics holistically to discern similarity and difference across frames. Such a small gate architecture is compatible with common edge devices, and it can be used as a plug-and-play module in CNN backbones to provide patch-skippable networks.

Efficient Federated Learning Framework on Heterogeneous Environment

Fri, 01 Apr 2022 00:00:00 +0000

Research Overview

Our team aims to design promising solutions for future AI applications in Federated Learning (FL) systems, which enable distributed computing nodes to collaboratively train machine learning models without exposing their own data. We focus on solving the following challenging issues:

Heterogeneous Hardware & Data
Resource constraints
Expensive communication
Lack of participants

Reference:

[1]. Edge Learning: the Enabling Technology for Distributed Big Data Analytics in the Edge. ACM Computing Surveys (TC), JCR-Q1

[2]. A Survey of Incentive Mechanism Design for Federated Learning. IEEE Transactions on Emerging Topics in Computational Intelligence (TETCI), JCR-Q1

Next generation blockchain system

Fri, 01 Apr 2022 00:00:00 +0000

Our team aims at the next-generation blockchain system with scalability, security, privacy, and intelligence and our proposed architecture is composed of 6 layers as above. In the following, the details of these 6 layers will be explained from top to bottom.

Radiation-free Spine Reconstruction and Posture Analysis Techniques with 3D Imaging

Sun, 27 Mar 2022 00:00:00 +0000

Scoliosis is a sideways curvature of the spine that occurs most often during thegrowth spurt just before puberty. According to the survey and statistics of China Child Development Center, more than 20% teens have scoliosis. In addition to myopia and obesity, scoliosis has become the third biggest killer of youth health. According to some relevant survey data, 60% of teenagers have different degrees of posture problems. Early prevention and early treatment is the key of adolescent posture problems. Children who have mild scoliosis need to be monitored closely, usually with X-rays,to see if the curve is getting worse.

At present, the mainstream screening method is still manual detection, which has require patient naturally stands and bends at 90 degrees and the doctor makes a judgment based on the spinal line and morphological changes. However, such method has low efficiency and low accuracy and facing the severe shortage of physiotherapists.

Meanwhile, during the COVID-19 epidemic, to help more people with rehabilitation needs realize remote screening and tracking and real-time AI monitoring of sports rehabilitation at home. A mobile-based posture screening algorithm and real-time exercise rehabilitation tracking algorithm are proposed.

In this project, we will design a 3D back image analysis method using commercial RGB-D to achieve low-cost, non-radiation, and sufficient accuracy of spine analysis and scoliosis screening. However, the analysis of the back image based on RGB-D is not simple and has the following three major challenges: (1) The back of the human body is an irregular curved surface, which is difficult to describe with some parametric models. (2) The point cloud image measured by commercial sensors is relatively sparse and has strong noise, and it is difficult to directly obtain the shape of the back. (3) The relationship between the back image and the spine shape is fuzzy.

Hardware Design and Compoments

To address these challenges, we propose an automated all-in-one machine that performs comprehensive and accurate analysis, evaluation, and diagnosis of human posture: analysis items are in addition to common body dimensions. It also includes three-dimensional reconstruction of human spine based on infrared, foot pressure analysis, XO legs, pelvic deformation analysis and other functions; it is a set of comprehensive posture evaluation system with clinical significance in the real sense.

Posture analysis algorithms

We also develop a Dr. Body APP, which uses AI algorithm to screen scoliosis, XO legs, high and low shoulders and other unhealthy postures through the photos of the back of the human body of the mobile phone for screening and severity classification; the use of a single photo of the human body for three-dimensional reconstruction of the human body for posture analysis and dimension measurement; Provide users with a variety of sports rehabilitation online courses, and record sports videos for intelligent action correction and analysis.

App record user data and provide personalized follow-up service

Innovation competition awards:

First Prize of Hong Kong University Student Innovation and Entrepreneurship Competition (2020); Hong Kong Social Enterprise Competition (HKSEC2019) Intellectual Property Ambassador Award; “Weining Cup” International AI Medical Challenge First Prize (2019), et.

Edge AI in smart city

Sun, 20 Mar 2022 00:00:00 +0000

Research Overview

Our team aims to design promising solutions for future AI applications based on edge intelligent technologies, which can empower construction, public health, environment, transportation and other industries, and promote the upgrading of urban intelligence.

COVID-19 Forward Forecast and Policy Evaluation Platform

Intelligent construction site safety detection system

Low-cost air quality testing technology

Edge Application Layer in Blockchain-empowered Edge Learning

Hybrid On-/Off-Chain Distributed Storage

Tue, 01 Mar 2022 00:00:00 +0000

Introduction

Personal data produced from widely emerged cyberspace activities are expected to promote information dissemination and engagement, or even make business intelligence more powerful. However, the recent increase in social media incidents of illegal surveillance and data breaches raises questions about the current data ownership model, in which centralized applications collect and control large amounts of user data. We present SocialChain, which is a decentralized online data storage and sharing system based on blockchain that decouples user data and applications to return data ownership to the user. We adopt Personal Data Store to extend off-chain storage for the online data, set up an identity establishment mechanism that can support WebID-based authentication functions using a unique identity assignment (i.e., WebID) as well as certificateless cryptography, and design a general framework that leverages smart contracts to help securely store and share social data in an automated manner.

Reference

SocialChain: Decoupling Social Data and Applications to Return Your Data Ownership, IEEE Transactions on Services Computing (CCF-B, JCR Q1), 2021.

Anti-Occlusion Human Pose Estimation for Scoliosis Rehabilitation

Sat, 01 Jan 2022 00:00:00 +0000

Physiotherapeutic scoliosis-specific exercises (PSSE) have been proved to be effective in scoliosis rehabilitation. PSSE consist of a program of curve-specific exercise protocols which are individually adapted to a patients’ curve site, magnitude, and clinical characteristics. Specific PSSE are used to help relieve scoliosis symptoms. PSSE is not generalized physiotherapy exercises which are generic exercises usually consisting of low impact stretching and strengthening activities like yoga, Pilates and the Alexander technique but especially designed exercise for scoliosis patients that requires professional guidance. Wrong movement may cause severe damage to the spine again.

During the COVID-19 pandemic, Exercise at specific medical centres with crowd around is risky. Smart phone, however, is now prevalence all over the world. Almost every smart phone contains at least one 2D camera that can capture video stream. If we can use this device to track patients’ movement, it may be way more convenient for doctors to guide the patients remotely.

Human pose estimation is one of the best matches of tracking patient movement at home.

Recent years have witnessed learning-based human pose estimation getting popular in more scenarios with occlusions, where invisible areas for perception objects significantly affect accuracy. Existing methods mainly use convolutional neural networks as the backbone and get limited local features to recover the occluded part. Such an anti-occlusion pipeline often suffers from the challenges of self-occlusion scenery, where similar parts of occluders and occludees are ambiguous. In this case, we need to design a masked visual autoencoder for image processing and video streaming, which recovers occluded regions by extracting deep spatial information at a higher semantic level. This autoencoder can get better details inferred from global self-attention and thus improves accuracy. The gist is to train the autoencoder to extract key-point information from the joint tokens that are manually masked in a self-supervised manner to simulate the occlusion in video streaming. To use dataset more efficiently and extract more information from a single image, we generate several different positive samples simulating more occluded cases by randomly mask more patches in joint tokens. We also propose an end-to-end pipeline for training and inference, which can effectively reduce the dependency of annotated occluded datasets and can be further applied to other visual tasks. This pipeline can obtain a great computation saving with much fewer annotated datasets and hold a higher runtime performance over the SOTA ViT methods.

Next Generation AI Video Analytics Detection System on Driving Behavior and Mental Factors

Mon, 13 Sep 2021 00:00:00 +0000

Video-based abnormal driving behavior and mental detection is becoming more and more popular. The key goal is to ensure the safety of drivers and passengers in the vehicle, and it is an essential step to realize autonomous driving at this stage.

Proposed Solution

Cognitive science, human facial expression, and eye tracking with AI Video Analytics Detection System.
To build up an AI model for evaluating the driving mental factors and driving behavior of driver to provide the alert signal for human-computer interaction.
Quantify Similarity of individual data to the Common Patterns Using Machine Learning Methods (i.e., log likelihoods of data being generated by models).

Methodology

Follow the previous research – In Psychology, HKU perform power analysis to determine the sample size (number of participants) required for detecting an effect, it have multiple predictors.
Mental factors features were identified in the previous research assuming a medium effect size (alpha 0.05 and 80% Power).
Big data collection in this project – Taxi Driver (at least 500 hours road driving video record) (e.g.: The Taxi driver who have been deducted the driving marks, in traffic jam situation).
Feature labelling - facial expression, texture, eye tracking, facial muscle change.
Train up the database.

Incentive Factors

Fanling Taxi Company (Mobile Shop Group Limited) supports this project, the camera can be installed in the Taxi to capture the real data (e.g.: angry or suddenly emotion change etc.) for analysis.
Mr. Wong (Director of Fanling Taxi Company) stated that his team want to join the Taxi Service Commendations Scheme and seek the technology to improve their quality of service. He and his team are to thirst for joining the trial.

Emergency Risk Management in Smart City

Sat, 27 Aug 2016 00:00:00 +0000

With the emergence and drastic improvement of mobile devices (e.g., phones, tablets, drones, and autonomous vehicles), we are now witnessing an exciting revolution of the digital city. Specifically, with the popularity of 5G networks, video-sharing and live-streaming applications (e.g., TikTok, BIGO, and Twitch) are becoming increasingly widespread. Besides allowing individuals to share life experiences and moments with their followers, video-based applications could also be a promising tool for sensing complex and dynamic urban environments.

With the development of AI-empowered video processing techniques, the ubiquitous mobile camera network has the potential to go beyond the video-surveillance system to capture various risks in the complex and dynamic city environment. However, the edge device has limited computation resource compared to the desktop GPU and CPU, which brings new challenge for optimizing AI algorithm for executing on edge devices.

Resource-constraint Mobile Devices

Another major challenge is the heterogeneous of edge hardware. The AI models need to adapt to heterogeneous edge devices with significant ram and computation difference. The last challenge is view limitation. Since each edge device usually has a small field of view, it is very hard to capture an integrated scenario in the smart city. To support practical real-time risk detection, we must joint use multiple cameras to obtain abundant regional data.

Heterogeneous Mobile Devices

To solve the above challenges, we proposed a crowdsourced mobile video analytics framework for emerging real-time risk management in smart cities. Specifically, our framework consists of three components, including city digital twins, collaborative learning mechanism and real-time risk management.

A Crowdsourced Mobile Video Analytics Framework

First, the digital twin is a replica of a physical word, and this pairing of the digital and physical worlds allows analysis of data and monitoring of systems to head off problems before they even occur.

Interactive Digital Twins for Smart City

Second, the high-performance collaborative learning mechanism can provide a fast and robust computation capacity to support the real-time requests of the big data analytics in smart cities.

High Performance Collaborative Learning

Third, derived from the above two techniques, we develop a real-time risk management platform for proactive risk detection and self-defending emergency response.

Real –time Risk Management Platform

The research team will conduct in-depth cooperation with professors from the Department of Real Estate and the Department of Electronic and Information Engineering to promote in-depth research on the security of construction sites based on risk management technology based on edge intelligence.

New Architectures and Methodologies for High Performance Sharding Blockchain

Wed, 27 Apr 2016 00:00:00 +0000

Blockchain draws tremendous attention from academia and industry, since it can provide distributed ledgers with data transparency, integrity, and immutability to untrusted parties for various decentralized applications. However, it is still challenging for blockchain to deal with large-scale networks because of the limited scalability of the blockchain systems. Sharding is a novel blockchain architecture that is proved to significantly improve the scalability of blockchain. Its main idea is to divide blockchain nodes into small groups called shards, which can handle transactions in parallel. To exploit the shortages of blockchain sharding for real application environment, we conduct two research projects as follows.

First, to deal with the serious performance degradation brought by cross-shard transactions, we propose a novel layered sharding architecture for blockchain and a cooperation-based layered sharding consensus. The basic idea is to allow shards to overlap, rather than isolating them completely, so that some nodes can locate in more than one shard. For cross-shard transactions, nodes located in the overlap of these shards can cooperative to verify, process and commit them directly and efficiently.

Blockchain and Training

Second, most existing blockchain sharding systems adopt a static sharding policy that cannot efciently deal with the dynamic environment in the blockchain system, i.e., joining and leaving of nodes, and malicious attack. We propose a novel dynamic sharding-based blockchain framework to achieve a good balance between performance and security without compromising scalability under a dynamic environment. For the framework, a deep reinforcement learning (DRL)-based consensus is designed to acquire optimal sharding policies in a series of dynamic and high-dimensional environment states.

Federated Learning in Resourced Constrained Mobile Edge Network

Fri, 01 Jan 2016 00:00:00 +0000

Federated learning (FL) has been proposed as a promising solution for future AI applications with strong privacy protection. It enables distributed computing nodes to collaboratively train models without exposing their own data. In this research topic, we focus on overcoming the heterogeneity challenge (e.g., data heterogeneity, model heterogeneity) in FL.

Collaborative Learning

Efficient Federated Learning on Heterogeneous Data: from the perspective of distribution characteristics of training data, FL can be categorized into two types, i.e., horizontal federated learning (HFL) and vertical federated learning (VFL). In HFL, we aim to propose efficient and robust learning scheme in resource-constrained computing environment by training a reinforcement learning model to adaptively tune the systematical parameters (e.g., the batch size in each client). Moreover, we explore the unbalanced features in VFL, the fundamental theories and algorithms are proposed to improve the learning efficiency and accuracy.

Knowledge Distillation

The Optimization of Federated Learning with Heterogenous Models: in this topic, we make effort on developing flexible and novel training framework by combining other techniques with FL, including (1) Knowledge Distillation (KD), (2) Generative Adversarial Networks (GAN), (3) Neural Architecture Search (NAS), (4) Meta Learning, etc. Furthermore, we investigate the personalization in FL, which is also a good way to handle the above challenges.

Projects | PEILab

A Unified TinyML System for Multi-modal Edge Intelligence and Real-time Visual Perception.

1. Introduction

2. Architecture Overview

3. Research Opportunities

4. Achievements

5. Related Publications

6. Cooperators

7. PhD/intern Applications:

Adaptive Quantization-aware Training and Model Compression.

Research Opportunity 1: Adaptive Quantization-aware Training and Model Compression

A Unified Contrastive Representation Learner for Cross-modal Federated Learning Systems.

Research Opportunity 3: A Unified Contrastive Representation Learner for Cross-modal Federated Learning Systems

Progressive Network Sparsification and Latent Feature Compression for Scalable Collaborative Learning.

Research Opportunity 4: Progressive Network Sparsification and Latent Feature Compression for Scalable Collaborative Learning

Masked Autoencoders for Occlusion-aware Visual Learners

Research Opportunity 5: Masked Autoencoders for Occlusion-aware Visual Learners

Flexible Patch Skip for Real-time Visual Perception.

Efficient Federated Learning Framework on Heterogeneous Environment

Research Overview

Reference:

Next generation blockchain system

Radiation-free Spine Reconstruction and Posture Analysis Techniques with 3D Imaging

Edge AI in smart city

Research Overview

COVID-19 Forward Forecast and Policy Evaluation Platform

Intelligent construction site safety detection system

Low-cost air quality testing technology

Edge Application Layer in Blockchain-empowered Edge Learning

Heterogeneous Data \& Resource Constraints- Batch Size Adaptation

Introduction

Reference:

Heterogeneous Data \& Expensive Communication- Layer-wised Aggregation

Introduction

Reference:

Semantic Query and Index Layer in Semantic Blockchain Database

Background

Reference

Heterogeneous Hardware \& Data- Parameterized Knowledge Transfer

Introduction

Reference:

Intelligent Consensus Layer in Learning-Driven Dynamic Architecture

Introduction

Reference

Lack of participants- Incentive Mechanism Design for Federated Learning

Introduction

Reference:

Layered Sharding Architecture for Blockchain

Introduction

Reference

Sustainable Off-chain Payment Channel Network

Introduction

Reference

Hybrid On-/Off-Chain Distributed Storage

Introduction

Reference

Anti-Occlusion Human Pose Estimation for Scoliosis Rehabilitation

Next Generation AI Video Analytics Detection System on Driving Behavior and Mental Factors

Proposed Solution

Methodology

Incentive Factors

Emergency Risk Management in Smart City

New Architectures and Methodologies for High Performance Sharding Blockchain

Federated Learning in Resourced Constrained Mobile Edge Network