Experiments: Add note, update script on removing W&B env var#2112
Experiments: Add note, update script on removing W&B env var#2112
Conversation
📚 Mintlify Preview Links📝 Changed (1 total)📄 Pages (1)
🤖 Generated automatically when Mintlify deployment succeeds |
🔗 Link Checker Results✅ All links are valid! No broken links were detected. Checked against: https://wb-21fd5541-experiments-shared-mode.mintlify.app |
| ```python | ||
| import wandb | ||
| import os | ||
|
|
||
| # Primary node | ||
| run = wandb.init( ... ) | ||
| os.environ['WANDB_RUN_ID'] = run.id | ||
|
|
||
| # Worker node | ||
| run = wandb.init( ... , id=os.environ['WANDB_RUN_ID']) | ||
| ``` | ||
| </Tip> |
There was a problem hiding this comment.
This example feels misleading to me because it looks like those statements run in the same script. I'm not sure this form is practical, since it requires somehow starting up worker nodes from the primary node's script to pass on the generated run ID.
The environment variable WANDB_RUN_ID is read automatically by wandb, and generally users should not read WANDB_ variables or use them for custom purposes. I also wouldn't advise anyone to modify os.environ at runtime, since changes might not be detected.
The essence of the original tip is to (1) generate your own ID and (2) set the WANDB_RUN_ID environment variable however you do it in the framework you're using to orchestrate the nodes. There's no general code sample we can provide as it's highly dependent on the user's setup.
| <Warning> | ||
| **Environment variable inheritance** | ||
|
|
||
| Worker processes may inherit W&B-related environment variables (such as `WANDB_SERVICE`) from parent processes in distributed environments. This can cause | ||
| initialization conflicts. | ||
|
|
||
| Clear inherited W&B environment variables (such as `WANDB_SERVICE`) before calling `wandb.init()` on worker nodes to avoid initialization conflicts: | ||
|
|
||
| ```python | ||
| import os | ||
| os.environ.pop('WANDB_SERVICE', None) | ||
| ``` | ||
| </Warning> |
There was a problem hiding this comment.
Oof, this one might be necessary in some cases, but that's definitely a bug on our side. Not sure about including this in the docs as it's a super janky thing to ask users to do.
2 participants
Description
Customers reported issue running code snippet on local, single machine. This was caused from W&B-specific env vars that were already set.
PR shows how to remove W&B env var (and which one). Also shows how to define run ID as an env var.
Some word smithing was done as well.
Related issues