How the X++ compiler works for AX2012

By request, today's blog attempts to explains the sequence of phases that are performed by the X++ compiler in AX 2012.

The compile process is a little bit involved, so a drawing quickly becomes quite complicated. Instead we've tried to present the explanation in two modes that compliment each other:
A. C#-ish pseudocode
B. Corresponding paragraphs of verbiage

C#-ISH PSEUDOCODE

// Phase1: Limited compile only for metadata
// (of class declarations and method signatures).

foreach(Element in AOT)
{
       Element.CompileHeaders();
}

// Phase2: Full compile for metadata and code bodies.

foreach(Element in AOT)
{
       compileSuceeded = Element.Compile();

       if (compileSuceeded == false)
       {
              ListOfBadElements.Add(Element);
       }
}

// Phase3 to PhaseN: Recompile the errored elements
// until success or error stabilization.

if (ListOfBadElements.Count > 0)
{
       while(true)
       {
              foreach(Element in ListOfBadElements)
              {
                     compileSuceeded = Element.Compile();
                     if (compileSuceeded == false)
                     {
                           NewListOfBadElements.Add(Element)
                     }
              }
             
              if (ListOfBadElements.Count == NewListOfBadElements.Count)
              {
                     break; // The list of elements with errors has been stabilized.
              }

              ListOfBadElements = NewListOfBadElements.GetCopy();
              NewListOfBadElements.Reset();
       }
}

VERBAL EXPLANATION

During a complete AOT compilation, there are at least two full compilation phases, plus the potential for N iterations of Phase3.

Phase1 is a pass over all the AOT elements. For now the compile operations concern only the class declarations and the method signatures. These limited compile operations create metadata about the types and their methods.
Reference errors are ignored during Phase1 because they are uninteresting consequences of the compile sequence. For example, the first class that is compiled might reference another class that has not quite yet been compiled, but which will be compiled a moment later in Phase1.

Phase2 is a pass over each AOT element to perform a full compile of each. The compiler validates all metadata, all references, and all X++ code. The compiler emits p-code (which the interpreter can later execute).
Any AOT element that suffers an error is added to a list of bad elements. In the explanatory pseudo code this list collection is named ListOfBadElements.

Phase3 occurs only if ListOfBadElements is not empty at the end of Phase2.
Phase3 is a pass through ListOfBadElements, so that each previously bad element can be recompiled. The elements that fail this recompile are added to a new list that is named NewListOfBadElements.
At the end of Phase3 there are three possible outcomes:

A. NewListOfBadElements.Count == 0
This means the overall compile has completed successfully. Error log will be empty. There might be warnings or informational messages (TODOs)

B. NewListOfBadElements.Count == ListOfBadElements.Count
This means the set of errors has stabilized (at a number above zero), which means the overall compile has failed. Error log will contain the errors, warnings and informational messages.

C. NewListOfBadElements.Count < ListOfBadElements.Count
This means that Phase3 made useful progress in reducing the count of errors. Therefore another phase like Phase3 must be run. This leads to Phase4, Phase5, and PhaseN for as many iterations as progress continues.
The more elements the AOT contains, the more likely it is that iterations beyond Phase3 will be needed.

AND THEN SOME...

Also note that the compile pass is doing more than crunching X++ code into executable p-code. It's performing a vast number of metadata validations and it's updating the security information by running the Security Inference process on applicable constructs to update the security information available.

Comments

• Anonymous
  June 12, 2013
  So, why can't the compiler be multi-threaded for the 1st pass? If it is just capturing metadata, why can't it capture it while running in more than 1 thread?
• Anonymous
  June 12, 2013
  @Michael Franchino:You are right, from an algorithmic perspective there are definitely options on the multi-threading side. The "easiest" one (not necessarily "easy" though due to intrinsic state that helps optimize the process already) is the first pass, which unfortunately isn't bringing vast improvements as it's the least of the work. If you look at the IL generation code in AX, you'll see some of this parallelism already in place.Retrofitting parallelism and making single-threaded implementations MT safe isn't a straight forward exercise. That's not the same as saying we're not looking at this though :)
• Anonymous
  June 12, 2013
  Thanks for sharing this. thumbs up :D
• Anonymous
  June 13, 2013
  Sounds good.. Compile has always been a real big pain, especially if the customer doesn't have the horsepower to do it.. Where I am working now, we have the compile running in about 45 minutes.. At my previous customer, it took 6 hours.Also, is there a way to have the compiler running on a separate thread as the client itself? Or having an external tool to do the compile so that at least you can see if it is still moving and where it is?
• Anonymous
  June 14, 2013
  @Michael Franchino: Stay tuned, we've got more improvements in the works along those lines :)
• Anonymous
  July 02, 2013
  The comment has been removed
• Anonymous
  July 30, 2013
  The comment has been removed