Share via

Static Compilation of IronPython scripts

  • Article

The ability to compile IronPython scripts into .NET IL and to save them to disk existed in IronPython 1.0 but has been missing in 2.0 so far. With IronPython 2.0 Beta4 this has been added back.

Why would I compile dynamic language scripts?

There are a lot of reasons to compile scripts into a binary form. Shri talks about some of them here. For folks who don't want to distribute source code in plain text this provides one level of obfuscation. To that end, a function called CompileModules has been added to the clr module to compile scripts into executable IL. The signature of the function is:

 CompileModules(str assemblyName, dict kwArgs, Array[str] filenames)

So to compile a file foo.py into foo.dll you would do this:

 import clr
clr.CompileModules("foo.dll", "foo.py")

This can now be brought in using the regular clr.AddReference. When clr.AddReference sees a compiled assembly, it publishes the module as well. So one can simply import the module into the code.

 clr.AddReference("foo.dll")
import foo

Multiple files and main

The function can take multiple python files and compile them into one dll. What if you want it to be a standalone executable? There are two things to be done. First, a stub exe is needed that can load the dll. Second, a way to distinguish the main module is needed. The keyword args that CompileModules can take comes in handy here

 import clr
clr.CompileModules("foo.dll", "foo.py", "bar.py", mainModule="main.py")

Now a stub exe can be written that loads up this compiled dll. The IronPython sample pyc.py has code that does shows how to generate a stub exe.

Wait, what is -X:SaveAssemblies mode then?

When IronPython is started with -X:SaveAssemblies, it generates a dll containing IL corresponding to the code it executed. Sounds an awful lot like compilation doesn't it? The difference is one is executable IL and the other isn't.

To understand the difference, one needs to understand that IronPython under normal course of its execution generates IL anyway. Every statement is converted to the DLR AST and IL gets spit out for the ASTs which is then executed. The SaveAssemblies mode simply dumps the generated IL into a dll. It is meant as a debugging device. So what is missing from this IL that prevents it from being re-executable code? The short answer is Dynamic Sites. The sites that are generated during the execution are not persisted. The compilation feature does exactly this - it persists the dynamic sites as well. Lets look at an example here and compare the generated IL in reflector. (Only the relevant code is copied over from reflector). This python code:

 print 2 + 5
print 2 * 5
print 3 / 5

when run with -X:SaveAssemblies mode produces this code:

 public static CallSite<DynamicSiteTarget<int, int, object>> #Constant207;
public static CallSite<DynamicSiteTarget<int, int, object>> #Constant208;
public static CallSite<DynamicSiteTarget<int, int, object>> #Constant209;

 $lineNo = 1;
PythonOps.Print(__global_context, #Constant207.Target(#Constant207, 2, 5));
$lineNo = 2;
PythonOps.Print(__global_context, #Constant208.Target(#Constant208, 2, 5));
$lineNo = 3;
PythonOps.Print(__global_context, #Constant209.Target(#Constant209, 3, 5));

Notice that the Constants defined here are actually defined as fields on the generated type and this type doesn't get instantiated anywhere and therefore the sites don't get assigned anywhere. The same python code when compiled with clr.CompiledModules produces this code:

 object[] objArray = new object[] { 
CallSite<DynamicSiteTarget<int, int, object>>.Create(PythonOps.MakeOperationAction(context, "Add")), 
CallSite<DynamicSiteTarget<int, int, object>>.Create(PythonOps.MakeOperationAction(context, "Multiply")), 
CallSite<DynamicSiteTarget<int, int, object>>.Create(PythonOps.MakeOperationAction(context, "Divide")) 
};

 line = 1;
PythonOps.Print(context, ((CallSite<DynamicSiteTarget<int, int, object>>)objArray[0]).Target(
    (CallSite<DynamicSiteTarget<int, int, object>>)objArray[0], 2, 5));
line = 2;
PythonOps.Print(context, ((CallSite<DynamicSiteTarget<int, int, object>>)objArray[1]).Target(
    (CallSite<DynamicSiteTarget<int, int, object>>)objArray[1], 2, 5));
line = 3;
PythonOps.Print(context, ((CallSite<DynamicSiteTarget<int, int, object>>)objArray[2]).Target(
    (CallSite<DynamicSiteTarget<int, int, object>>)objArray[2], 3, 5));

You can see that all the dynamic call sites are being created here and their targets are being invoked. This then is perfectly executable code - maybe not as succinct as the python code but it does the same thing :)

Comments

  • Anonymous
    August 05, 2008
    PingBack from http://blog.a-foton.ru/2008/08/static-compilation-of-ironpython-scripts/

  • Anonymous
    October 20, 2008
    The comment has been removed

  • Anonymous
    October 23, 2008
    The comment has been removed

  • Anonymous
    October 24, 2008
    The comment has been removed

  • Anonymous
    October 25, 2008
    Yes I checked and pyc can't be used with 2.0 version since the only attributes available in Hosting call are: ['ErrorCodes', 'Python', 'PythonCommandLine', 'PythonConsoleOptions', 'PythonOptionsParser'] So how is the independent executable made after all?

  • Anonymous
    October 26, 2008
    Are you using the 1.0 pyc sample by any chance? The sample was updated for 2.0 and is here - http://www.codeplex.com/IronPython/Release/ProjectReleases.aspx?ReleaseId=14353

  • Anonymous
    October 27, 2008
    Thanks for your help ;) I managed to compile it with pyc but I must've misunderstand the topic since when I compiled it still requires python lib and Ironpython dlls. am I doing something wrong or it's suppose to be that way? I mean to a simple app to distribute I have to included the all of python lib? It's not pratical, I must be wrong.

  • Anonymous
    October 27, 2008
    You will require the four IronPython DLLs for the exe to run but you can compile the python libraries along with your py files. It doesn't have to be the entire python library as well- only files that are required by your app.

  • Anonymous
    October 27, 2008
    So i need to include in the compile all the regular modules? os, sys, socket, so on? But sometimes they need other modules to run. For instance. When i compiled the last time it required me some modules and other modules those modules run. Isn't there a easy way to know wich one are the sub-modules?

  • Anonymous
    October 31, 2008
    There might be some static dependency analysis tools for python out there. I'm not aware of any though.

  • Anonymous
    December 10, 2008
    The comment has been removed

  • Anonymous
    December 22, 2008
    C:TemppyProg.exe should work since the dependencies are picked up from the directory of the executable as well. If you want to copy pyprog.exe to some other place, you can create a pyprog.exe.config file and use the assemblybinding tag - http://msdn.microsoft.com/en-us/library/twy1dw1e(VS.80).aspx

  • Anonymous
    March 04, 2009
    Hi, we are migrating an existing cpython app to ironpython to work with silverlight. We have so many packages and subpackages in cpython and to work with silverlight we tried to convert them all into a dll. The dll conversion is fine and all files are part of dll, but all the packages can be referred only from the main module. The modules that are present inside packages are not able to refer any other packages. Are we missing something basic.

  • Anonymous
    March 08, 2009
    Estuve investigando el supuesto camino de pre-compilar m&oacute;dulos para ver si lograba resolver los problemas de performance que mencion&eacute; hace unos d&iacute;as. Resulta que la punta del iceberg estaba escondida en el pack de ejemplos de ...

  • Anonymous
    March 09, 2009
    Dinesh, packages importing other packages from within the dll should work without any extra work. If you can send me a minimal reproduction of the problem, I can probably be of more assitance. Feel free to use the contact link to e-mail me - http://blogs.msdn.com/srivatsn/contact.aspx

  • Anonymous
    April 11, 2010
    The comment has been removed

  • Anonymous
    April 12, 2010
    StringIO is part of the standard library. So you should include stringio.py as well (and its dependencies) while compiling a.dll

  • Anonymous
    April 12, 2010
    Thank you, this helped me a bit further. Now I am running in to a new error which has to do with code access security [ADO.NET]: SystemError: Fehler bei der Anforderung des Berechtigungstyps System.Data.SqlClient.SqlClientPermission, System.Data, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089. Is there any easy way to grant the "a.dll" all the rights which IronPython has?