paperFL

FL相关论文库

Categories 分类：

Artificial Intelligence 人工智能 (IJCAI, AAAI, AISTATS, ALT, AI)

联邦学习论文被以下顶级人工智能会议和期刊接受，包括：

  • IJCAI (International Joint Conference on Artificial Intelligence)
  • AAAI (AAAI Conference on Artificial Intelligence)
  • AISTATS (Artificial Intelligence and Statistics)
  • ALT (International Conference on Algorithmic Learning Theory)
  • AI (Artificial Intelligence)

Machine Learning 机器学习 (NeurIPS, ICML, ICLR, COLT, UAI, Machine Learning, JMLR, TPAMI)

联邦学习论文被以下顶级机器学习会议和期刊接受，包括：

  • NeurIPS (Annual Conference on Neural Information Processing Systems)
  • ICML (International Conference on Machine Learning)
  • ICLR (International Conference on Learning Representations)
  • COLT (Annual Conference on Computational Learning Theory)
  • UAI (Conference on Uncertainty in Artificial Intelligence)
  • Machine Learning
  • JMLR (Journal of Machine Learning Research)
  • TPAMI (IEEE Transactions on Pattern Analysis and Machine Intelligence)

Data Mining 数据挖掘 (KDD, WSDM)

联邦学习论文被以下顶级数据挖掘会议和期刊接受，包括：

  • KDD (ACM SIGKDD Conference on Knowledge Discovery and Data Mining)
  • WSDM (Web Search and Data Mining)

Secure 安全 (S&P, CCS, USENIX Security, NDSS)

联邦学习论文被以下顶级安全会议和期刊接受，包括：

  • S&P (IEEE Symposium on Security and Privacy)
  • CCS (Conference on Computer and Communications Security)
  • USENIX Security (Usenix Security Symposium)
  • NDSS (Network and Distributed System Security Symposium)

Computer Vision 计算机视觉 (ICCV, CVPR, ECCV, MM, IJCV)

联邦学习论文被以下顶级计算机视觉会议和期刊接受，包括：

  • CVPR (Computer Vision and Pattern Recognition)
  • ICCV (IEEE International Conference on Computer Vision)
  • ECCV (European Conference on Computer Vision)
  • MM (ACM International Conference on Multimedia)
  • IJCV (International Journal of Computer Vision)

Natural Language Processing 自然语言处理 (ACL, EMNLP, NAACL, COLING)

联邦学习论文被以下顶级自然语言处理会议和期刊接受，包括：

  • ACL (Annual Meeting of the Association for Computational Linguistics)
  • NAACL (North American Chapter of the Association for Computational Linguistics)
  • EMNLP (Conference on Empirical Methods in Natural Language Processing)
  • COLING (International Conference on Computational Linguistics)

Information Retrieval 信息检索 (SIGIR)

联邦学习论文被以下顶级信息检索会议和期刊接受，包括：

  • SIGIR (Annual International ACM SIGIR Conference on Research and Development in Information Retrieval)

Database 数据库 (SIGMOD, ICDE, VLDB)

联邦学习论文被以下顶级数据库会议和期刊接受，包括：

  • SIGMOD (ACM SIGMOD Conference)
  • ICDE (IEEE International Conference on Data Engineering)
  • VLDB (Very Large Data Bases Conference)

Network 网络 (SIGCOMM, INFOCOM, MOBICOM, NSDI, WWW)

联邦学习论文被以下顶级网络会议和期刊接受，包括：

  • SIGCOMM (Conference on Applications, Technologies, Architectures, and Protocols for Computer Communication)
  • INFOCOM (IEEE Conference on Computer Communications)
  • MobiCom (ACM/IEEE International Conference on Mobile Computing and Networking)
  • NSDI (Symposium on Networked Systems Design and Implementation)
  • WWW (The Web Conference)

System 系统 (OSDI, SOSP, ISCA, MLSys, EuroSys, TPDS, DAC, TOCS, TOS, TCAD, TC)

联邦学习论文被以下顶级系统会议和期刊接受，包括：

  • OSDI (USENIX Symposium on Operating Systems Design and Implementation)
  • SOSP (Symposium on Operating Systems Principles)
  • ISCA (International Symposium on Computer Architecture)
  • MLSys (Conference on Machine Learning and Systems)
  • EuroSys (European Conference on Computer Systems)
  • TPDS (IEEE Transactions on Parallel and Distributed Systems)
  • DAC (Design Automation Conference)
  • TOCS (ACM Transactions on Computer Systems)
  • TOS (ACM Transactions on Storage)
  • TCAD (IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems)
  • TC (IEEE Transactions on Computers)

Others 其他 (ICSE, FOCS, STOC)

联邦学习论文被以下其他领域的顶级会议和期刊接受，包括：

  • ICSE (International Conference on Software Engineering)
  • FOCS (IEEE Annual Symposium on Foundations of Computer Science)
  • STOC (Symposium on the Theory of Computing)

System

Title Affiliation Venue Year Materials
DeTA: Minimizing Data Leaks in Federated Learning via Decentralized and Trustworthy Aggregation IBM Research EuroSys 2024 PUB
FLOAT: Federated Learning Optimizations with Automated Tuning Virginia Tech EuroSys 2024 PUB CODE
Totoro: A Scalable Federated Learning Engine for the Edge UCSC EuroSys 2024 PUB
Dordis: Efficient Federated Learning with Dropout-Resilient Differential Privacy HKUST EuroSys 2024 PUB CODE
FLIGAN: Enhancing Federated Learning with Incomplete Data using GAN EuroSys workshop 2024 PUB
ALS Algorithm for Robust and Communication-Efficient Federated Learning EuroSys workshop 2024 [x]
FedRDMA: Communication-Efficient Cross-Silo Federated LLM via Chunked RDMA Transmission EuroSys workshop 2024 PUB
REFL: Resource-Efficient Federated Learning QMUL EuroSys 2023 PUB CODE
A First Look at the Impact of Distillation Hyper-Parameters in Federated Knowledge Distillation EuroSys workshop 2023 PUB
Towards Practical Few-shot Federated NLP EuroSys workshop 2023 PUB
Can Fair Federated Learning Reduce the need for Personalisation? EuroSys workshop 2023 PUB
Gradient-less Federated Gradient Boosting Tree with Learnable Learning Rates EuroSys workshop 2023 PUB
Towards Robust and Bias-free Federated Learning EuroSys workshop 2023 [x]

Tuning

Title of the Paper Authors Link Skimming/In-depth Reading
CPET: Effective Parameter-Efficient Tuning for Compressed Large Language Models Weilin Zhao, Yuxiang Huang, Xu Han, Zhiyuan Liu, Zhengyan Zhang, Maosong Sun [Arxiv] S
Star ReMax: A Simple, Effective, and Efficient Method for Aligning Large Language Models Ziniu Li, Tian Xu, Yushun Zhang, Yang Yu, Ruoyu Sun, Zhi-Quan Luo [ICML 2024] [Github] S
TRANSOM: An Efficient Fault-Tolerant System for Training LLMs Baodong Wu, Lei Xia, Qingping Li, Kangyu Li, Xu Chen, Yongqiang Guo, Tieyao Xiang, Yuheng Chen, Shigang Li [Paper]
DEFT: Data Efficient Fine-Tuning for Large Language Models via Unsupervised Core-Set Selection Devleena Das, Vivek Khetan [Paper]
Star LongQLoRA: Efficient and Effective Method to Extend Context Length of Large Language Models Jianxin Yang [Paper] [Github]
Star Sparse Fine-tuning for Inference Acceleration of Large Language Models Eldar Kurtic, Denis Kuznedelev, Elias Frantar, Michael Goin, Dan Alistarh [Paper] [Github] [Github]
Star ComPEFT: Compression for Communicating Parameter Efficient Updates via Sparsification and Quantization Prateek Yadav, Leshem Choshen, Colin Raffel, Mohit Bansal [Paper] [Github]
Towards Better Parameter-Efficient Fine-Tuning for Large Language Models: A Position Paper Chengyu Wang, Junbing Yan, Wei Zhang, Jun Huang [Paper]
Star SPT: Fine-Tuning Transformer-based Language Models Efficiently with Sparsification Yuntao Gui, Xiao Yan, Peiqi Yin, Han Yang, James Cheng [Paper] [Github]
Star LoRA+: Efficient Low Rank Adaptation of Large Models Soufiane Hayou, Nikhil Ghosh, Bin Yu [Paper] [Github]
Sparse MeZO: Less Parameters for Better Performance in Zeroth-Order LLM Fine-Tuning Yong Liu, Zirui Zhu, Chaoyu Gong, Minhao Cheng, Cho-Jui Hsieh, Yang You [Paper]
Star DropBP: Accelerating Fine-Tuning of Large Language Models by Dropping Backward Propagation Sunghyeon Woo, Baeseong Park, Byeongwook Kim, Minjung Jo, Sejung Kwon, Dongsuk Jeon, Dongsoo Lee [Paper] [Github]
LoRA-SP: Streamlined Partial Parameter Adaptation for Resource-Efficient Fine-Tuning of Large Language Models Yichao Wu, Yafei Xiang, Shuning Huo, Yulu Gong, Penghao Liang [Paper]
Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey Zeyu Han, Chao Gao, Jinyang Liu, Jeff (Jun) Zhang, Sai Qian Zhang [Paper]
Publish Star AILS-NTUA at SemEval-2024 Task 6: Efficient model tuning for hallucination detection and analysis Natalia Griogoriadou, Maria Lymperaiou, Giorgos Filandrianos, Giorgos Stamou [Paper] [Github]
Star BAdam: A Memory Efficient Full Parameter Training Method for Large Language Models Qijun Luo, Hengxu Yu, Xiao Li [Paper] [Github]
Intuition-aware Mixture-of-Rank-1-Experts for Parameter Efficient Finetuning Yijiang Liu, Rongyu Zhang, Huanrui Yang, Kurt Keutzer, Yuan Du, Li Du, Shanghang Zhang [Paper]
Publish Star Random Masking Finds Winning Tickets for Parameter Efficient Fine-tuning Jing Xu, Jingzhao Zhang [Paper] [Github]
Zeroth-Order Fine-Tuning of LLMs with Extreme Sparsity Wentao Guo, Jikai Long, Yimeng Zeng, Zirui Liu, Xinyu Yang, Yide Ran, Jacob R. Gardner, Osbert Bastani, Christopher De Sa, Xiaodong Yu, Beidi Chen, Zhaozhuo Xu [Paper]
Publish Star Light-PEFT: Lightening Parameter-Efficient Fine-Tuning via Early Pruning Naibin Gu, Peng Fu, Xiyu Liu, Bowen Shen, Zheng Lin, Weiping Wang [Paper] [Github]
BlockLLM: Memory-Efficient Adaptation of LLMs by Selecting and Optimizing the Right Coordinate Blocks Amrutha Varshini Ramesh, Vignesh Ganapathiraman, Issam H. Laradji, Mark Schmidt [Paper]
Compress then Serve: Serving Thousands of LoRA Adapters with Little Overhead Rickard Brüel-Gabrielsson, Jiacheng Zhu, Onkar Bhardwaj, Leshem Choshen, Kristjan Greenewald, Mikhail Yurochkin, Justin Solomon [Paper]
Publish Star Increasing Model Capacity for Free: A Simple Strategy for Parameter Efficient Fine-tuning Haobo Song, Hao Zhao, Soumajit Majumder, Tao Lin [Paper] [Github]
Publish PocketLLM: Enabling On-Device Fine-Tuning for Personalized LLMs Dan Peng, Zhihui Fu, Jun Wang [Paper]
Code Less, Align More: Efficient LLM Fine-tuning for Code Generation with Data Pruning Yun-Da Tsai, Mingjie Liu, Haoxing Ren [Paper]

CPET: Effective Parameter-Efficient Tuning for Compressed Large Language Models

  • 机构: 清华大学 NLP Group, DCST, IAI, BNRIST
  • 问题：以前的模型在进行模型压缩时可能会导致知识丢失和性能下降，尤其是在大型语言模型（LLMs）上，这种压缩可能会影响模型在下游任务上的表现。
  • 优势：CPET模型的优势在于它结合了参数高效调整（PET）和模型压缩技术，通过知识继承和恢复策略来弥补压缩过程中可能导致的性能下降。
  • 解决方案的关键在于引入了两种机制：(1) PET知识继承，使用在非压缩LLM上训练的PET模块作为初始化来学习压缩LLM上的PET模块；(2) 模型知识恢复，通过在压缩LLM中添加额外的知识恢复模块来弥补压缩过程中丢失的知识。

ReMax: A Simple, Effective, and Efficient Method for Aligning Large Language Models

  • 机构:
    • 香港中文大学（深圳）数据科学学院
    • 深圳大数据研究所
    • 南京大学国家新型软件技术重点实验室
    • Polixir.ai
    • 黄埔帕洲实验室（中国深圳国际工业与应用数学中心）
  • 问题：以前的模型，特别是PPO，对于LLMs来说过于复杂，需要大量的计算资源和内存，且超参数调整繁琐，导致在有限的计算资源下难以应用。
  • 优势：ReMax模型之所以好，是因为它简化了实现，减少了超参数，降低了GPU内存使用，并缩短了训练时间。它利用了RLHF的三个特性：快速模拟、确定性转换和轨迹级奖励，这些在PPO中未被充分利用。
  • 关键：ReMax算法的关键是在REINFORCE算法的基础上引入了一种新的方差减少技术，并且去除了PPO算法中的价值模型部分，从而简化了算法，减少了内存消耗和训练时间。

TRANSOM: An Efficient Fault-Tolerant System for Training LLMs

  • 机构:
    • SenseTime，Huazhong University of Science and Technology
    • Beijing University of Posts and Telecommunications
  • 问题：以前的模型在训练过程中容易受到硬件和软件故障的影响，导致训练任务中断，需要耗费大量时间和资源进行任务重启和恢复。此外，传统的方法在检查点保存和加载时效率较低，增加了整体训练时间
  • 优势：
    TRANSOM通过设计三个关键子系统（TOL、TEE、TCE），实现了高效的容错和恢复机制。这些机制显著提升了大规模LLMs训练的效率，减少了训练中断的时间和资源消耗。例如，GPT3-175B的预训练时间减少了28%，任务重启时间从数小时减少到12分钟，检查点保存和加载时间从平均200秒减少到不到10秒 。
  • 论文中的解决方案之关键是设计了三个关键子系统：
    1. 自动容错和恢复机制的训练管道（TOL）
    1. 多维度度量自动异常检测系统（TEE）
    1. 训练检查点异步访问自动容错和恢复技术（TCE）

DEFT-UCS: Data Efficient Fine-Tuning for Pre-Trained Language Models via Unsupervised Core-Set Selection

  • 机构:
  • Georgia Institute of Technology
  • Accenture Labs
  • 问题：以前的模型在微调过程中需要大量高质量数据，增加了数据获取和处理的成本。传统微调方法虽然能达到较好的性能，但在实际应用中往往面临数据不足的问题
  • 优势：DEFT-UCS通过无监督核心集选择方法，实现了在减少数据量的情况下高效微调预训练语言模型。实验结果表明，DEFT-UCS在多个文本编辑任务中，使用70%的训练数据可以达到与使用全部数据的传统方法相当的性能。
  • 关键：无监督核心集选择（UCS）方法。UCS通过K-Means聚类选择代表性数据子集，从而减少微调所需的数据量

LongQLoRA: Efficient and Effective Method to Extend Context Length of Large Language Models

  • 机构: Sun Yat-sen University
  • 问题：论文试图解决大型语言模型（如LLaMA2）在有限的训练资源下，如何有效地扩展其上下文长度的问题。当前许多模型在处理超出预定义上下文长度的输入时，性能显著下降，导致在处理长文本任务（如多文档问答、书籍总结等）时表现不佳。
  • 优势：它在单个32GB V100 GPU上高效地扩展了大型语言模型（LLaMA2）的上下文长度，同时保持了良好的性能。这一方法结合了位置插值（Position Interpolation）、QLoRA和Shift Short Attention的优点，实现了计算资源的节约和性能的提升。
  • 关键：结合了位置插值（Position Interpolation）、QLoRA和Shift Short Attention。位置插值用于扩展上下文长度，QLoRA通过量化预训练模型权重和添加可学习的低秩适配器来节省GPU内存，而Shift Short Attention通过分组计算注意力来进一步节省计算资源。

Sparse Fine-tuning for Inference Acceleration of Large Language Models

  • 机构：IST Austria
  • 问题：以前的模型在高稀疏度下微调时，容易出现训练不稳定的问题，导致准确性下降。此外，现有的量化方法在进一步压缩时（如到3比特）会导致准确性难以恢复，限制了推理速度的提升。
  • 优势：因为它能够在高效运行的同时保持高准确性。本文提出的SquareHead蒸馏方法能够在高稀疏度下进行微调，既能加速推理速度，又能保持模型的准确性，这使得模型在实践中更有用。
  • 关键：论文中解决方案的关键在于提出了一种基于L2的蒸馏方法SquareHead，通过这种方法在高稀疏度下进行微调时，能够有效地恢复模型的准确性。同时，展示了利用稀疏性实现推理加速的方法。

ComPEFT: Compression for Communicating Parameter Efficient Updates via Sparsification and Quantization

  • 研究机构:
  1. University of North Carolina at Chapel Hill
  2. IBM Research
  3. Massachusetts Institute of Technology
  4. University of Toronto
  5. Vector Institute
  • 问题：以前的PEFT模型虽然在参数高效微调上表现良好，但在高延迟网络环境中传输较大模型时存在明显的通信成本问题。此外，传统压缩方法在不进行额外训练的情况下，通常会导致模型性能的显著下降。
  • 优势：ComPEFT模型好是因为它通过稀疏化和三值量化显著减少了PEFT模型的大小，同时在多个任务和模型上的性能都保持或有所提升。压缩后的模型在高延迟网络中的传输更为高效，且在少样本组合泛化能力上表现出色。
  • 关键：关键点在于稀疏化和三值量化相结合，使得任务向量在大幅度压缩的同时仍能保持高性能。特别是选择合适的标量常数（α）用于量化，有助于恢复或提高模型性能。

SPT: Fine-Tuning Transformer-based Language Models Efficiently with Sparsification

  • 机构:
    • The Chinese University of Hong Kong, Hong Kong SAR, China
    • Southern University of Science and Technology, China
  • 问题：
    • 高内存消耗：存储多头注意力机制的权重需要大量内存。
    • 计算成本高：前馈网络的计算成本高，导致运行时间长。
    • 微调效率低：直接微调大型预训练模型需要更新大量参数，效率较低。
  • 优势：SPT通过引入稀疏化技术，在不显著降低模型质量的情况下，减少了内存消耗和计算成本，显著提升了微调Transformer模型的效率
  • 关键：稀疏化技术：
    • 稀疏MHA：通过只计算和存储前L个注意力权重，减少了内存消耗。
    • 路由FFN：通过动态激活每个token的一部分模型参数，减少了计算成本。

LoRA+: Efficient Low Rank Adaptation of Large Models

  • 机构: Simons Institute, UC Berkeley, Department of Statistics, UC Berkeley
  • 问题：以前的 LoRA 模型在处理大宽度模型时效果不佳，因为它为适配器矩阵 A 和 B 设置相同的学习率，导致特征学习效率低下，微调效果不理想。
  • 优势：LoRA+ 通过为 LoRA 适配器矩阵 A 和 B 设置不同的学习率，提高了特征学习效率，使得大宽度模型在微调时能够更有效地学习和适应新任务，从而在性能和速度上都有显著提升。
  • 关键：解决方案的关键是为 LoRA 适配器矩阵 A 和 B 设置不同的学习率，并通过缩放理论确定一个合适的固定比例。这种方法被称为 LoRA+，它在相同计算成本下提高了特征学习效率和模型性能。

LLM