Debugger stuck trying to load variables

[akhileshraju]

Environment data

• debugpy version: 1.6.0
• OS and version:
  • VS Code UI running on Win 10 Enterprise - Build 18363
  • Development env - Ubuntu 18.04 (Connected via the Remote SSH plugin)
• Python version: Python 3.10
• Using VS Code

Actual behavior

When trying to debug python code that is part of python package chipscopy, the debugger in VS Code gets stuck trying to resolve variables.

Once this happens, no other debug functionality works. The only option left is to use the "stop" button and end the debugging session.

Here's is the launch config I am using

{
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Engine",
            "type": "python",
            "request": "launch",
            "cwd": "${workspaceFolder}",
            "linux": {
                "python": "${workspaceFolder}/.venv/bin/python"
            },
            "windows": {
                "python": "${workspaceFolder}\\.venv\\Scripts\\python.exe"
            },
            "program": "${workspaceFolder}${pathSeparator}engine${pathSeparator}main.py",
            "env": {
                "PYTHONPATH": "${workspaceFolder}${pathSeparator}engine:${PYTHONPATH}",
                "DEBUGPY_LOG_DIR": "${workspaceFolder}${pathSeparator}engine"
            },
            // https://github.com/microsoft/vscode-python/issues/13719#issuecomment-705768580
            "rules": [
                {"module": "chipscopy", "include": true},
                {"module": "chipscopy.tcf", "include": false}
            ],
            "justMyCode": true,
            "preLaunchTask": "python:run-protoc",
            // Ctrl-C cannot be sent using debug console
            "console": "integratedTerminal"

        }
    ]
}

Expected behavior

I expect debugging to work just fine since debugging the same application in PyCharm works without issues.

Steps to reproduce:

I, unfortunately, cannot share the source code since it's proprietary. I have attached the debugpy logs.
Please let me know if I can provide any other info.

debugpy.pydevd.5433.log
debugpy.server-5433.log

[int19h]

The debugger invokes repr() on any types it doesn't specifically know about. We try to do it in a safe manner, but an implementation of __repr__ still has the potential to crash the Python runtime, so I suspect that this is what might be happening here. Is there any particular type that is present in Variables that is triggering it for you?

@fabioz, is there anything we can do from the logging perspective to make this easier to diagnose, i.e. which specific variables failed to repr? Looking at the logs, all we have is the "variables" request that didn't come back. Also, one thing that is interesting in the logs is that there are two "stackTrace" requests in rapid sequence, and the "variables" request that crashed refers to the frame for the first "stackTrace" response, but is issued after the second "stackTrace" request is received and processed.

[akhileshraju]

Sorry couldn't get back to you sooner. Thanks for that hint @int19h.
Yes, one of the local variables that I am trying to introspect does define a custom __repr__. The code in this __repr__ does reach out over the IO and read information from hardware.

When I commented out the __repr__ in question, things seem to work at first. But when I try to introspect the device variable, it again goes back into the 'forever spinning' state, probably because there are other attributes in the device class that are instances of classes with custom __repr__.

Is there a way I could tell the debugger to not go searching for __repr__'s and invoke them?
Ideally, I dont know if it's possible, I would expect the debugger to simply move on it the repr leads to bad behavior, much like what I assume PyCharm does. But at this point, any help to make the debugger usable would be much appreciated.

[int19h]

The problem is that it's impossible to tell in advance whether a given implementation of __repr__ is going to take much time, and the API doesn't offer any opportunities for cooperative cancellation. The best we could do is call it on a background thread and inject an exception after timeout - but even then, if it gets stuck in a native call (like I/O), a Python exception wouldn't be able to interrupt it. On some platforms, at least, threads can be forcibly killed from the outside - but that also means that resources don't get cleaned up, and, more importantly, that any locks the killed thread held remain such, which will result in hard-to-diagnose deadlocks.

I think the best we can do here is roughly what you suggested - some option that would control our strategy on using or not using repr to begin with. Right now, we use it for most types, but provide our own custom fallbacks for some standard types and third-party libraries like numpy. We could have a mode where this is flipped - a list of types for which built-in or our custom repr is allowed, and generic "instance of type ..." for everything else.

[fabioz]

@akhileshraju maybe an option here with the current structure is create a pydevd plugin yourself (if you know the class the debugger allows you to create an extension and customize how you want the repr to work for that class -- i.e.: since you know the variable which is showing the spinner you can customize how it shows).

For details see:
https://github.com/microsoft/debugpy/tree/main/src/debugpy/_vendored/pydevd/pydevd_plugins/extensions

And there are some real-word examples in:
https://github.com/microsoft/debugpy/tree/main/src/debugpy/_vendored/pydevd/pydevd_plugins/extensions/types

[akhileshraju]

The problem is that it's impossible to tell in advance whether a given implementation of __repr__ is going to take much time, and the API doesn't offer any opportunities for cooperative cancellation. The best we could do is call it on a background thread and inject an exception after timeout - but even then, if it gets stuck in a native call (like I/O), a Python exception wouldn't be able to interrupt it. On some platforms, at least, threads can be forcibly killed from the outside - but that also means that resources don't get cleaned up, and, more importantly, that any locks the killed thread held remain such, which will result in hard-to-diagnose deadlocks.

I think the best we can do here is roughly what you suggested - some option that would control our strategy on using or not using repr to begin with. Right now, we use it for most types, but provide our own custom fallbacks for some standard types and third-party libraries like numpy. We could have a mode where this is flipped - a list of types for which built-in or our custom repr is allowed, and generic "instance of type ..." for everything else.

I see @int19h . Thanks for explaining the issue.
If I am understanding it correctly, the option to control the debugger's repr invocation strategy isn't already in the debugger right?

@akhileshraju maybe an option here with the current structure is create a pydevd plugin yourself (if you know the class the debugger allows you to create an extension and customize how you want the repr to work for that class -- i.e.: since you know the variable which is showing the spinner you can customize how it shows).

For details see: https://github.com/microsoft/debugpy/tree/main/src/debugpy/_vendored/pydevd/pydevd_plugins/extensions

And there are some real-word examples in: https://github.com/microsoft/debugpy/tree/main/src/debugpy/_vendored/pydevd/pydevd_plugins/extensions/types

If no debugger out there worked, it would be a different issue at hand. While VS Code has been great all this while for me and still remains my preferred IDE, since PyCharm seems to handle this exact scenario just fine, unfortunately I dont think I can afford to spend time on writing a plugin.

To test the theory that the repr might be the source of the issue, I commented out all the repr's from the package in question. While that got me a bit farther, now it hangs trying to resolve I don't know which attribute in the device.

The logs look a bit different now -- I don't know if the issue is somewhere else now.

debugpy.pydevd.39671.log
debugpy.server-39671.log

[int19h]

Yep, there's no such option at present. My suggestion was basically to make this issue into such an enhancement request.

[int19h]

As to why the problem persists - aside from __repr__, there are other magic functions that have to be used to enumerate the variables in the first place and retrieve their values. Thus, custom implementations of e.g. __getattr__, __iter__, __getitem__ etc can manifest a similar problem, as does the use of @property. This is the unfortunate side effect of Python's extreme dynamism - since virtually everything about an object can be redefined in ways that might throw or hang, there are very few operations on objects that are intrinsically safe, like comparing references or retrieving the object's type.

(With exceptions, at least, we can and do proactively catch them from anything that could possibly be running code that does that, even if it's extreme corner cases.)

[akhileshraju]

Hmm I see.

@fabioz's suggestion to define a custom plugin, that wouldn't help with things other than __repr__ issues right?

[fabioz]

One thing that occurred is that we show more info than PyCharm itself, so, maybe you can try filtering the same way.

To do that, in your launch configuration put:

        "variablePresentation": {
            "all": "hide",
            "protected": "inline",
        },

as an option (this would make it show less things -- possibly one of those that'd no longer be shown could be the culprit for crashing).

I'll take a look into improving the logging so that it's at least easier to diagnose where exactly it's getting stuck.

[akhileshraju]

I tried your suggestion @fabioz but the debugger is still stuck ☹

For reference, here is my launch config with the snippet you provided

{
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Engine",
            "type": "python",
            "request": "launch",
            "cwd": "${workspaceFolder}",
            "linux": {
                "python": "${workspaceFolder}/.venv/bin/python"
            },
            "windows": {
                "python": "${workspaceFolder}\\.venv\\Scripts\\python.exe"
            },
            "program": "${workspaceFolder}${pathSeparator}engine${pathSeparator}main.py",
            "env": {
                "PYTHONPATH": "${workspaceFolder}${pathSeparator}engine:${PYTHONPATH}",
                // "DEBUGPY_LOG_DIR": "${workspaceFolder}${pathSeparator}engine"
            },
            "justMyCode": false,
            "preLaunchTask": "python:run-protoc",
            // Ctrl-C cannot be sent using debug console
            "console": "integratedTerminal",
            "variablePresentation": {
                "all": "hide",
                "protected": "inline",
            },
        }
    ]
}

[int19h]

A custom plugin can also define custom logic to retrieve attributes of an object of some type, so it could skip any expensive properties, or wrap them into something that only shows the value when expanded.

Unfortunately, that is the best that we can do here.