Skip to content

fix: remove dead reference to non-existent module .file_utils#1164

Open
anzzyspeaksgit wants to merge 1 commit into
flagos-ai:mainfrom
anzzyspeaksgit:fix/issue-1137
Open

fix: remove dead reference to non-existent module .file_utils#1164
anzzyspeaksgit wants to merge 1 commit into
flagos-ai:mainfrom
anzzyspeaksgit:fix/issue-1137

Conversation

@anzzyspeaksgit
Copy link
Copy Markdown

Summary

Removed the dead reference to the non-existent .file_utils module in gpt2_tokenization.py as requested.

Problem

Closes #1077

Solution

The from_pretrained method in GPT2Tokenizer previously attempted to import cached_path from .file_utils (a remnant of old huggingface/transformers code). Because .file_utils does not exist in the flagscale.train.megatron.training.tokenizer module, it could cause ImportError / EnvironmentError. I removed the try-except block entirely to keep the tokenizer loading strictly local vocabulary files cleanly.

Testing

  • Verified pytest does not break by missing imports.

This PR was created with AI assistance.

@CLAassistant
Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Dead reference to non-existent module

2 participants