Adding IronPython scripting engine to Mdbg
I hear IronPython is a great managed scripting language to embed in other managed apps, so I thought I'd try this out by writing an MDbg Extension to drop IronPython 0.9.1 into the MDBg Beta 2 sample. (both pieces are publicly available downloads). The experiment went pretty well. I conclude IronPython is a very viable choice for scripting managed apps, I can already do cool things, and I'm excited at what I could do if I actually knew how to use Python.
I'll blog about this in a 2-parter: First is a focus on my actual results (the demo), and next I'll focus on how easy it was to get there. [update:] I wrote up the steps for how-to add IronPython support.
I also need to emphasize that I had never used python before this and am not on any python forums. I used https://diveintopython.org/toc/index.html as a basic python reference and got some tips from Martin Maly. More comprehensive reference is https://docs.python.org/tut. I can see that the python gurus could do a lot of amazing things here.
All of the Python-Mdbg work was done in a single Mdbg extension and didn't require any modifications to either Python or Mdbg. In fact, my python extension even operates well with my Mdbg gui extension. This is very promising because if your existing managed app already has a plug-in model, you may be able to write a single Python plug-in that can then turn around and run arbitrary python scripts.
The C# source for my python extension is here. I also have a sample collection of my python scripts for mdbg here, which contains the python functions I mention below. The extension adds the following Mdbg commands:
[Update: 11/29/05] Note that this is an Mdbg extension, which means that you must compile the extension (source here) and then load it into Mdbg (via the "Load") command before you can use any of the commands below. Eg, if you compile the extension into "PythonExt.dll", then you can load it into Mdbg via "load pythonext.dll".
1) "python" command executes a single python statement. Mdbg still owns the input console, so this is what forwards input from the Mdbg command prompt onto the python engine. This is the bread and butter.
mdbg> py 1+2
3[p#:0, t#:0] mdbg> py Shell
<Microsoft.Samples.Tools.Mdbg.MDbgShell object at 0x021A74D4>[p#:0, t#:0] mdbg> py PrintStack()
0) Foo.Second (test.cs:35)
1) Foo.Main (test.cs:29)
------------
The first case is a simple expression. The second case shows that the python interpreter has access to the Shell object, from which it can use reflection to access the rest of Mdbg. In the third case, we execute the python function PrintStack(), which I defined as:
def GetDebugger(): return Shell.Debugger; def CurProc(): return GetDebugger().Processes.Active def CurThread(): return CurProc().Threads.Activedef PrintStack(): i = 0 for x in CurThread().Frames: print "%d) %s" % (i, x) i = i + 1 print '------------'
You can imagine that you could create powerful python commands to dump arbitrary data structures.
2) The "pyin" (python interactive) command enters a python interactive session similar to the standard python console. This lets you issue a bunch of python commands (useful for multiple-line stuff) without needing the mdbg "python" command before each one. The console input gets directly routed to the python engine input loop instead of to MDbg's input loop. You exit back to an Mdbg session by typing "pyout".
[p#:0, t#:0] mdbg> pyin
Entering Python Interactive prompt.
Type 'pyout' leave PythonInteractive mode and get back to Mdbg prompt.
>>>
[p#:0, t#:0] mdbg> 1+2
3
>>>
[p#:0, t#:0] mdbg> PrintVal(CurLocals())
CS$4$0000=False(System.Boolean)
>>>
[p#:0, t#:0] mdbg> pyout
Leaving Python Interactive prompt.
[p#:0, t#:0] mdbg>
PrintVal() and CurLocals() are also functions I defined in Python, similar to PrintStack() above:
def CurLocals(): return CurFrame().Function.GetActiveLocalVars(CurFrame())def PrintVal(v): # Check for MDbgValue[] if str(type(v))=="<type 'Microsoft.Samples.Debugging.MdbgEngine.MDbgValue[]'>": for x in v: PrintVal(x) else: PrintValHelper(v, "") # Helper to print with indent. def PrintValHelper(v, indent): if (v.IsComplexType): print indent + v.Name + "=Complex Type: (" + v.TypeName + ")" for x in v.GetFields(): PrintValHelper(x, indent+" ") elif (v.IsArrayType): print indent + v.Name + "=array Type: (" + v.TypeName + ")" for x in v.GetArrayItems(): PrintValHelper(x, indent+" ") else: print indent + v.Name + "=" + v.GetStringValue(0, False) + "(" + v.TypeName + ")";
3) The "pimport" (Python Import) command loads a python module (some file with a .py extension that contains the same raw stuff you'd type into the console) into the default scope. This is key because it lets you build up libraries of all your great functions (like PrintVal, PrintStack, etc) in some script file. It turns out that python has an "import" command, so you may be tempted to issue an MDbg command like "python import test". However, imported python modules won't have visibility to Mdbg variables like "Shell", and so they can't do interesting Mdbg stuff. I needed a dedicated mdbg import command to import the python script into the right scope.
4) The "Pyb" (Python Breakpoint) command which lets you attach a python expression to a breakpoint. In general, I think there's a lot of power from being able to use python expressions as predicates / filters throughout your app. In this case, if the expression returns a non-null result, that becomes the StopReason and Mdbg stops at the breakpoint. If the expression evaluates to null, the breakpoint is continued.
You can use this to implement conditional breakpoints.
The pyb syntax is: "pyb" <breakpoint location> "|" <python expression>
In the following example, we set a python conditional breakpoint inside the for loop. E2() is yet another python function I wrote which fetches the value of a local variable and returns it as a string.
[p#:0, t#:0] mdbg> sh 6 29 Second(5); 30 Console.WriteLine("Hoo!"); 31 Second(4); 32 } 33 34 static void Second(int z) 35:*{ 36 Console.WriteLine("second"); 37 int x; 38 for(x = 5; x < 8; x++) 39 Console.WriteLine(x); <-- We'll set breakpoint here 40 } [p#:0, t#:0] mdbg> pyb 39 | E2("x")=="6" <-- Stop on line 39 when local expression "x" evaluates to "6". Breakpoint #3 bound (line 39 in test.cs)(python: E2("x")=="6") [p#:0, t#:0] mdbg> b Current breakpoints: Breakpoint #1 bound (Foo::Second(+0)) Breakpoint #3 bound (line 39 in test.cs)(python: E2("x")=="6") <-- See our breakpoint shows up in the list [p#:0, t#:0] mdbg> g Python BP hit: E2("x")=="6" Result:False (of type=System.Boolean) <-- First iteration, x==5 Expression is false. continuing Python BP hit: E2("x")=="6" Result:True (of type=System.Boolean) <-- Second iteation, x==6 Expression is true. Stopping at breakpoint STOP True 39: Console.WriteLine(x); [p#:0, t#:0] mdbg> print x x=6 [p#:0, t#:0] mdbg>
Some ways you could use Python in Mdbg:
There are lots of ways that you could wire up Mdbg with Iron Python script. For example:
1) fancy conditional breakpoint: You could have multiple breakpoints invoking different methods on the same python class and manipulating the same data. For example, you could imagine building some sort of compound breakpoint that had tentacles in multiple methods like AddRef() and Release(), did stack hashing at each hit, and helped track down ref leaks. Or you could build an "instance" breakpoint that creates a strong handle to a given instance and then sets conditional breakpoints on all member functions which compare the current instance to the cached strong-handle. The possibilities are endless.
2) fancy exception filtering: Only want to stop on 1st chance exceptions thrown from a given module while function Foo() is on the stack, but function Bar() is not? Write a python expression to hook Mdbg's Exception notification callback.
3) texual visualizers: Need to dump the contents of your own hand-rolled collection? Take the PrintStack() example above a few steps further and you could write python script to inspect any arbitrary data structures. You could even dump it as HTML to a file to get a rich view.
4) mass breakpoints: You want to place a breakpoint on every single method described by a regular expression? Trivial! (Python handles regular expressions).
In conclusion: For not knowing anything about python when I first started, this went very smoothly. The entire extension (source is here) is verbosely commented with lots of whitespace and still under 450 lines. I also found several Mdbg bugs and have recognized some things MDbg could be doing to cooperate with being more scriptable.
The thing that excites me the most is how scalable the solution is. Writing your own scripting engine is hard. Using the existing IronPython engine is easy and powerful. The python snippets can be arbitrary python expressions (including function calls with side-effects) and can share data with each other. If an app embeds the IronPython engine, it allows end users to easily wire things up in new and useful ways.
[update] Next up, I'll blog about the actual mechanics of how I went about hooking this up.
Comments
- Anonymous
August 31, 2005
Very nice Post. - Anonymous
September 01, 2005
Previously, I added IronPython scripting support to a real existing application, MDbg (a managed debugger... - Anonymous
September 02, 2005
Mike Stall recently completed a project to embed IronPython into the MDbg debugger as an MDbg extension.&nbsp;... - Anonymous
September 02, 2005
Mike Stall recently completed a project to embed IronPython into the MDbg debugger as an MDbg extension.&nbsp;... - Anonymous
September 02, 2005
Mike Stall recently completed a project to embed IronPython into the MDbg debugger as an MDbg extension.&nbsp;... - Anonymous
September 02, 2005
Mike Stall recently completed a project to embed IronPython into the MDbg debugger as an MDbg extension.&nbsp;... - Anonymous
September 03, 2005
nice post. - Anonymous
September 07, 2005
Shawn Farkas just pointed out I have abug in the MyStream::Write where for 'r'. Whoops!
I'll updated the source with his suggested fix. - Anonymous
August 25, 2006
A while ago, we had a sample of Mdbg-Python extension that let you use python to script MDbg. It was...