2024

We introduce an LLM decoding acceleration method that requires no fine-tuning. Our approach involves an adaptive draft-verification process that evolves over time to improve efficiency. We utilize a tri-gram matrixbased LLM representation to dynamically approximate the output distribution of the LLM, allowing the model to adjust to changing token probabilities during the decoding process.

We introduce ToolNet, a plug-and-play method to assists LLMs in handling massive tools. ToolNet organizes tools in a weighted directed graph (node represents tools and edges denote tool transition) based on the tool-use trajectories produced by LLMs. An LLM navigates in the graph by iteratively choosing the next one from its successors until the task is resolved. Graphs are updated online, enabling adjustment to accommodate the frequent updates of tools and new tasks.

We introduce a distribution-aligned method for black-box LLM-generated text detection. It aligns the surrogate model’s distribution with the unknown target LLMs. It enriches widely adopted zero-shot detection methods (DNA-GPT, Fast-DetectGPT, etc.) with a ‘plug-and-play’ enhancement feature.

We propose an adaptive computational method that dynamically allocates computation resources in each sampling step to improve the generation efficiency of diffusion models. Our method can be seamlessly integrated into any existing diffusion models (both CNN- and Transformer-based) and can be easily combined with approaches that reduce the number of sampling steps in diffusion models.

We develop a five-session interactive course on ChatGPT's features, limitations, prompt engineering techniques, ethical considerations, and strategies for incorporating ChatGPT into teaching. Our thematic analysis highlights that after the course, teachers have a more positive & nuanced understanding of ChatGPT, with quotes with positive connotations rising from 45% to 68%.

We have developed the first network-oriented LLM to master an end-to-end wireless optimization simulator. To enable LLM-driven simulation mechanism, we create an instruction-following dataset containing Q&A pairs generated from intricate tutorial documents. Using Sionna as a case study, we employ novel joint parameter efficient fine-tuning and retrieval-augmented generation techniques to adapt the generic LLMs into network-oriented LLMs.

We introduce D-REC, a DT-assisted reliable RL mechanism for wireless caching optimization. Unlike existing approaches, D-REC incorporates on-demand constraints, including state, reward, and action safety modules, to prioritize network reliability and sustainability.

We propose EQ-ViT, an end-to-end acceleration framework with algorithm-architecture co-design features to enable real-time ViT acceleration on AMD Versal Adaptive Compute Acceleration Platform.

We comprehensively investigate value alignment approaches. We first unpack the historical context of alignment tracing back to the 1920s (where it comes from), then delve into the mathematical essence of alignment (what it is), shedding light on the inherent challenges.

We present a modular prompting framework to solve robustness issues in code comprehension for LLMs by leveraging main-purpose reasoning guidance and iterative reasoning enhancement.

We explore cost-effective radio map generation using generative diffusion probabilistic models, applicable to both indoor and outdoor wireless network scenarios, particularly valuable in complex scenarios where obtaining comprehensive measurements is challenging.

We investigate for the first time the reliability of sparse training from an out-of-distribution (OOD) perspective, which jointly considers OOD reliability and efficiency and has important implications for real-world deep neural network applications.

We present PGExplainer, a parameterized explainer for Graph Neural Networks (GNNs). PGExplainer adopts a deep neural network to parameterize the generation process of explanations and provide a global understanding of any GNN models on arbitrary machine learning tasks.

We propose an adaptive gradient correction method to accelerate and stabilize sparse training. Our method reduces the number of epochs up to 52.1% compared to the leading sparse training methods. Our method is compatible with both unstructured and structured sparse training pipelines.

We introduce AutoST, a training-free NAS method for Spiking Transformers, to rapidly identify high-performance Spiking Transformer architectures.

This study makes contributions to the field of computer science education, and explores the feasibility of utilizing large language models (LLMs) for automating feedback for Java programming assignments in an introductory computer science (CS1) class.

2023

We present a modular ALM framework to solve multi-step reasoning by decoupling reasoning from tool feedback and observations. Theoretical decomposition of prompt tokens establishes that our method substantially reduces prompting redundancy in prevailing Thought-Action-Observation ALM systems.

We present an augmented language model platform that enables flexible customization of agents through simple configurations, seamlessly integrating various language models, task formats, prompting modules, and plugins into a unified paradigm.

We propose two model augmentation techniques, i.e. using early-stage models and weight perturbation to learn an informative synthetic set with significantly reduced training cost. Extensive experiments demonstrate that our method achieves up to 20× speedup.

We propose a novel early exiting strategy based on cascading input similarity with valid assumptions on saturation states in visual-language models, a pioneering exploration of extending early exiting selection to encoders and decoders of sequence-to-sequence architectures.

We show that distilled data lead to not-calibratable networks due to the loss of information that is semantically meaningful but unrelated to classification tasks. We propose Masked Temperature Scaling & Distillation Training to mitigate these limitations while maintaining the efficiency.

This paper focuses on addressing the practical yet challenging problem of model heterogeneity in federated learning, where clients possess models with different network structures. We propose pFedHR, focusing on solving the problem of heterogeneous model cooperation.

We aim to answer: (i) What is the core to defend against adversarial attacks for sparse language models? (ii) How can we efficiently prevent the loss of pre-trained knowledge in pruning to preserve or even enhance robustness?

This paper introduces controllable randomness by generating binary masks in a specific random fashion. We aim to answer: (i) Which is better for pruning? a deterministic way or a randomized way? (ii) Can we design a consistently effective randomized pruning method?

We propose a circular training framework, Colead, which co-evolves both the data-driven synthesizer and the NLU-driven synthesizer to achieve high-quality code generation in the presence of data scarcity and domain growth.

We introduce EfficientLight, an RL-based traffic signal control method that balances model size and performance. In multi-intersection scenarios, our method outperforms all baseline methods with the lowest #paras and the smallest computational cost compared to other RL-based methods.

We develop an adaptive access point (AP) planning approach that can accurately sense the environment dynamics, reconstruct the obstacle map, and then predict the placements of mmWave APs adaptively.

This study highlights the pedagogical significance of predicting useful comments in mutual assessment to promote student learning and reduces the need to collect labeled data via domain adaptation.

How can we quantify the uncertainty in the learning process and enable reliable rare category analysis? We jointly learn the characterizations of rare categories and calibrate the confidence.

We walk through the ODQA models and conclude the core techniques on efficiency. Quantitative analysis on memory cost, processing speed, accuracy and overall comparison are given.

We for the first time identify and study the reliability problem of sparse training and find that sparse training exacerbates the over-confidence problem of DNNs. We then develop a new sparse training method, CigL, to produce more reliable sparse models.

This study explores how ChatGPT personalizes the learning experience, how it can be augmented with math and physical performance, and how educators can ensure that the LLM algorithm is unbiased.

To assist explainable sparse training, we propose important weights exploitation and weights coverage exploration to characterize sparse training. Our method does not need to train dense models, achieving up to 95% sparsity ratio and even higher accuracy than dense training, with same amount of iterations.

We propose an energy efficient spiking neural network training workflow, and design a new drop-andgrow strategy with decreasing number of non-zero weights in the process of dynamically updating sparse mask. We demonstrate extremely high sparsity (i.e., 99%) model performance in SNN based vision tasks.

We develop a gradient-like proposal for any discrete distribution without this strong requirement. Built upon a locally-balanced proposal, our method efficiently approximates the discrete likelihood ratio via a Newton’s series expansion to enable a large and efficient exploration in discrete spaces.

We propose a class-based covariance transfer method from the perspective of disentangling to transfer covariance information in long-tailed classification task.

We propose an autonomous image composition and masking method for cloud masking, a fundamental task in Earth observation problems across social sectors such as agriculture, energy, and water.

We propose an adaptive data augmentation method to avoid ad-hoc choices or painstakingly trial-and-error tuning for time series representation learning.

2022

We develop a few-shot task-agnostic NAS framework, AutoDistil, for distilling large language models into compressed students with variable computational cost. AutoDistil outperforms leading baselines with upto 3x additional reduction in computational cost and negligible loss in task performance.

We introduce the first commercial hardware platform supporting high-degree sparsity acceleration up to 32 times — S4. S4 provides a (sparse) equivalent computation power of 944 TOPS in INT8 and 472 TFLOPS in BF16, and has 20GB LPDDR4 memory with up to 72 GB memory bandwidth in a low 70 Watt power envelope. We demonstrate several-times practical inference speedup on S4 over mainstream inference platforms such as Nvidia T4.

We propose AE-BERT, an automatic and efficient pruning framework. AE-BERT achieves the inference time of a single BERT-BASE encoder on Xilinx Alveo U200 FPGA board that is 1.83x faster compared to Intel(R) Xeon(R) Gold 5218 (2.30GHz) CPU.

We study network pruning of Transformer-based language models under the pre-training and fine-tuning paradigm and propose a counter-traditional hypothesis that pruning increases the risk of overfitting when performed during the fine-tuning phase.

2021

We propose an information-aware contrastive learning framework for graph-structure data, and show for the first time that all recent graph contrastive learning methods can be unified by our framework.

We study how knowledge is transferred and lost during the pre-train, fine-tune, and pruning process, and propose a knowledge-aware sparse pruning process that achieves significantly superior results than existing literature.

We study data augmentation for cross-lingual natural language inference and propose two methods of training a generative model to induce synthesized examples to reflect more diversity in a semantically faithful way.

We utilize multi-instance learning to model the uncertainty of precursor period, and design a contrastive loss to address the issue that annotated anomalies are few.

We propose MT-RMN to dynamically learn task relationships and accordingly learn to assemble composable modules into complex layouts to jointly solve multiple sequence processing tasks.

We propose TRRN to model temporal networks by employing transformer-style self-attention to reason over a set of memories.

We propose MoveSD to model state transition in human movement from a novel perspective, by learning the decision model and integrating the system dynamics.

We introduce Longitudinal deep kernel Gaussian process regression to fully automate the discovery of complex multi level correlation structure from longitudinal data.

2020

We propose to adopt deep neural networks to parameterize the generation process of explanations, which enables a natural approach to multi-instance explanations.

We propose a semi-supervised adversarial perturbation framework that encourages the model to be more robust towards such divergence and better adapt to the target language.

We propose a deep architecture for learning trends in multivariate time series, which jointly learns both local and global contextual features for predicting the trend of time series.

We propose longitudinal kulti-level factorization machine, to the best of our knowledge, the first model to address these challenges in learning predictive models from longitudinal data.

2019

We propose an adaptive neural network for node classification in dynamic networks, which is able to consider the evolution of both node attributes and network topology.

We propose a spatio-temporal attentive RNN model, which aims to learn node representations for classification by jointly considering both the temporal and spatial patterns of the node.

DeepCC utilizes the deep autoencoder for dimension reduction, and employs a variant of Gaussian mixture model to infer the cluster assignments. A mutual information loss is proposed to bridge the training of instances and features.

2018

DMNE coordinates multiple neural networks (one for each input network data) with a co-regularized loss function to manipulate cross-network relationships, which can be many-to-many, weighted and incomplete.

Most multi-instance learning (MIL) methods that study true positive instances ignore 1) the global similarity among positive instances and 2) that negative instances are non-i.i.d.. We propose a MTL method based on positive instance graph updating to address this issue.

This paper reviews the research progress of multi-instance learning (MTL), introduces different assumptions, and categories MTL methods into instance-level, bag-level, and embedded-space. Extensions and major applications in various areas are discussed at last.

2017

We propose a self-adaptive locality-sensitive hashing encoding method for multi-instance learning (MIL), which efficiently deals with large MIL problems.

We propose a metric learning method for multi-instance classification, aiming to find an instance-dependent metric by maximizing the relative distance on neighborhood level.

2016

We propose a positive instance detection method based on multiple instance learning, of which the core idea is that true positive instances should not only be similar to themselves globally but also different from negative instances robustly.

We propose a metric learning approach called multi-metrics classification machine. We establish an optimization problem for each class (each metric) to learn multiple metrics independently.

2015

We introduce the definition of clustering, the basic elements involved in clustering process, and categorize the clustering algorithms into the traditional ones and the modern ones. All the algorithms are discussed comprehensively.

Undergraduate

This paper proposes a support vector machine-based ensemble model to forecast crude oil price based on VECM and stochastic time effective pattern modelling and recognition system (STEPMRS).

This paper presents an integrated model to forecast crude oil prices, where pattern modelling & recognition system is used to model the price trend and error correction model is offered to forecast errors. A neural network layer is employed to integrate the results.

*Last updated on 9/27/2024*