bug


说在前面

面对huggingface被墙的情况，我想了很多种办法：

  • 1.Linux服务器中设置代理
  • 2.将模型下载到本地在上传到服务器上
  • 3.使用huggface镜像站(神中神！！强烈推荐)

由于最后一种方法是在太简便，所以我们这里直介绍最后一种方法


参考链接

还是和往常一样贴几个参考链接：


问题

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

Traceback (most recent call last):
  File "/cyb/LAVIS/train.py", line 108, in <module>
    main()
  File "/cyb/LAVIS/train.py", line 97, in main
    model = task.build_model(cfg)
  File "/cyb/LAVIS/lavis/tasks/base_task.py", line 33, in build_model
    return model_cls.from_config(model_config)
  File "/cyb/LAVIS/lavis/models/blip2_models/blip2_opt.py", line 412, in from_config
    model = cls(
  File "/cyb/LAVIS/lavis/models/blip2_models/blip2_opt.py", line 85, in __init__
    self.opt_tokenizer = AutoTokenizer.from_pretrained(opt_model, use_fast=False)
  File "/root/anaconda3/envs/ab/lib/python3.9/site-packages/transformers/models/auto/tokenization_auto.py", line 667, in from_pretrained
    config = AutoConfig.from_pretrained(
  File "/root/anaconda3/envs/ab/lib/python3.9/site-packages/transformers/models/auto/configuration_auto.py", line 983, in from_pretrained
    config_dict, unused_kwargs = PretrainedConfig.get_config_dict(pretrained_model_name_or_path, **kwargs)
  File "/root/anaconda3/envs/ab/lib/python3.9/site-packages/transformers/configuration_utils.py", line 617, in get_config_dict
    config_dict, kwargs = cls._get_config_dict(pretrained_model_name_or_path, **kwargs)
  File "/root/anaconda3/envs/ab/lib/python3.9/site-packages/transformers/configuration_utils.py", line 672, in _get_config_dict
    resolved_config_file = cached_file(
  File "/root/anaconda3/envs/ab/lib/python3.9/site-packages/transformers/utils/hub.py", line 452, in cached_file
    raise EnvironmentError(
OSError: We couldn't connect to 'https://huggingface.co' to load this file, couldn't find it in the cached files and it looks like facebook/opt-2.7b is not the path to a directory containing a file named config.json.

尝试从Hugging Face Hub下载facebook/opt-2.7b模型失败了，具体代码位于blip2_opt.py文件的初始化方法中，调用AutoTokenizer.from_pretrained(opt_model, use_fast=False)时出现的。


解决方法

安装依赖

1
pip install -U huggingface_hub

设置环境变量

Linux：

1
export HF_ENDPOINT=https://hf-mirror.com

Windows Powershell,对于有梯子的用户应该不用使用此方法就可以下载:

1
$env:HF_ENDPOINT = "https://hf-mirror.com"

同时建议，将上面这一行写入 ~/.bashrc中，这样就不需要每次都输入该指令了。

下载模型

1
huggingface-cli download --resume-download gpt2 --local-dir gpt2
  • --resume-download:后面填模型地址
  • --local-dir：后面填指定路径(可以不填，默认.~/cache/huggingface/hub)

下载数据集

1
huggingface-cli download --repo-type dataset --resume-download wikitext --local-dir wikitext

可以添加 --local-dir-use-symlinks False 参数禁用文件软链接，这样下载路径下所见即所得，详细解释请见上面提到的教程。


后记

这是/cyb下的目录：

1
2
3
4
.
├── LAVIS
├── apt-cache
└── opt-2.7b

我之前一直不懂opt-2.7b是什么用，现在才发现就是上面我需要下载的那个facebook/opt-2.7b,我也是刚理解的学姐的这番话(所以其实我不用下载对吧…废了这么大劲全走弯路了T.T)