Evaluating Iron scripting languages for use as a test framework
Not so long ago, I was passing through the hallway and someone from my team was giving praises about the XML based scripting language they were using to automate some tests. This was particularly desirable because the process of compiling test changes, copying binaries to the correct location on the virtual machine, and rerunning the test is a bit tedious. With scripting the XML file could be edited on the VM and the working file could be copied back to the development machine and checked in.
My reaction to this was to wonder why we needed a new scripting language when there were already so many in existence! This is especially true considering each component probably would require its own XML processing engine so there are probably multiple out there.
One additional benefit to using scripting for your tests is that since scripts are often interpreted, your build times should be reduced over implementing tests in C#. But there are also some down sides:
- Detecting API changes and incompatibilities won’t happen until runtime when using scripting languages.
- You may have to install additional binaries to run your scripts
- Many people are probably not proficient enough in a scripting language such that they would consider using it for testing. It takes time to learn a new language.
- Hosting a scripting language requires additional learning and research if you require a mixture of compiled and scripted code.
In this post, I will show some examples of scripts to run a simple scenario using the Unified Communications Managed SDK (UCMA), which I’ve used once or twice before. I will attempt this in Iron Python 2.7 and Iron Ruby 1.1.3.
Note: PowerShell is another possible choice, but there are some additional things to overcome so I hope to handle it in a future post.
Challenges
Each language has some strengths and some challenges when it comes to dealing with existing .Net code. In particular, the APIs we are testing have to have equal or better usability for the following C# code:
endpoint.RegisterForIncomingCall<InstantMessagingCall>(this.CallReceived);
call.StateChanged += this.CallStateChanged;
Basically, the scripting languages need to deal with passing generic delegates, and registering event handlers that use an EventHandler<TArgs> signature.
Using Iron Python
I am more familiar with Python than Ruby, so out of the gate Python has an advantage and it may be visible in my writing in this post. If you are fluent in Iron Ruby and would like to point out some corrections to put Ruby on a more even playing field, feel free to leave a comment at the bottom of the post.
Iron Python has built in support for generics, which I happened to know when writing the sample scripts. This helped a lot in making the RegisterForIncomingCall usage easy. This section of code is shown here:
print 'Registering for incoming call'
self.__endpoint.RegisterForIncomingCall[InstantMessagingCall](self.__IMCallReceived)
Also in Iron Python, registering for event handlers is similar to C# and doesn’t require much of a learning curve:
self.__imCall.StateChanged += self.__CallStateChanged
But to load a dll so you can use it is a bit unintuitive at first:
import clr
import sys
sys.path.append('<path to dll>')
clr.AddReference("Microsoft.Rtc.Collaboration.dll")
and then python requires importing the symbols you need. I did it the hard way:
from Microsoft.Rtc.Collaboration import ServerPlatformSettings,CollaborationPlatform,ApplicationEndpointSettings,ApplicationEndpoint
from Microsoft.Rtc.Collaboration import Conversation,InstantMessagingCall,CallEstablishOptions
from Microsoft.Rtc.Signaling import FailureResponseException
it could probably be simplified a bit like this:
from Microsoft.Rtc.Collaboration import *
from Microsoft.Rtc.Signaling import *
but it is probably the case you would pull in too many type names as .Net probably is not as conservative on what is exposed as Python libraries are.
The rest of the issues with python are the usual preference items that come up when comparing languages. For example, you will see “self” repeated a lot in python classes, and the space significance of the blocks may not be to your liking.
Using Iron Ruby
Not being too familiar with Ruby, I had to do a little research to get started. After picking up the Ruby necessities my first challenge was getting the reference set up to the API I was interested in testing. After trying a few things, I was able to get the API to test loaded, but it probably not the best way:
require 'C:\windows\microsoft.net\assembly\GAC_MSIL\Microsoft.Rtc.Collaboration\<version>\Microsoft.Rtc.Collaboration.dll'
include Microsoft::Rtc::Collaboration
Note that it is being included directly from the GAC. I should be able to specify the assembly information and it would hopefully find it in the GAC, but that didn’t seem to work unfortunately.
The next thing was to figure out the RegisterForIncomingCall invocation. This is what I came up with:
# The C# equivalent of these two lines is:
#
# endpoint.RegisterForIncomingCall<InstantMessagingCall>(this.IMCallReceived)
#
p = Proc.new { |sender,e| IMCallReceived(sender,e) }
@endpoint.method(:RegisterForIncomingCall).of(InstantMessagingCall).call(p)
As you can see, this is probably not the most optimal way to do it in Ruby. A small interactive experiment shows that Ruby does support generic syntax:
>>> require 'System'
=> true
>>> include System::Collections::Generic
=> Object
>>> x = List[System::Int32].new
=> []
>>> x.add(4)
=> nil
>>> x
=> [4]
>>> x.add(7)
=> nil
>>> x
=> [4, 7]
>>>
but that is how I initially got it working. I probably could go back and change the code now, but finding the better way to register the event handlers is really bugging me. It seems that this should have worked:
# The following should also be valid, but doesn't seem to work. Because of generics?
#
# @imCall.InstantMessagingFlowConfigurationRequested.add do |sender, e|
# FlowConfigurationRequested(sender,e)
# end
But it didn’t work for me as you can see from the comment. I eventually got it to work as follows:
# In C#: call.InstantMessagingFlowConfigurationRequested += this.FlowConfigurationRequested
p = Proc.new { |sender,e| FlowConfigurationRequested(sender,e) }
@imCall.InstantMessagingFlowConfigurationRequested.add p
Although it works, it isn’t really intuitive to someone coming from C#. Maybe I can do something with the += operator to make this easier, but out of the box it didn’t seem to allow that. If you have experience with this in Iron Ruby feel free to leave a pointer in the comments on what the best way to register the event handlers.
Other than that, everything is pretty straight forward and again, the only thing left are tidbits related to language syntax and preferences. Ruby is not sensitive to whitespace to determine its block boundaries like Python is, but you need to be sure to match up your “ends” correctly.
Ruby has more support for creating DSLs (sub languages) based on Ruby syntax which I am interested in trying out at some point. With a little more time spent with Ruby I’ll hopefully be able to pick up the best practices and improve my example.
Choosing your language
Given that some language aspects are likely to bug people and affect their choice of languages, it is very likely that both languages will need to be available in the test environment. Unfortunately, Iron Python and Iron Ruby currently use different versions of the hosting API which makes having them both present somewhat tricky.
The recommended solution is to download the source and recompile so they use the same hosting APIs. This may be a barrier to entry that is too high for some people and drive them back to inventing their own scripting languages again. I quickly attempted this, but it will take more time and investigation to get it working.
Summary
In this post I presented two existing scripting languages that people could try instead of rolling their own. Below I’ll include the full text of the implementations so you can see more clearly. Hopefully you will be inspired to spend one or two days learning a scripting language and consider using an existing one rather than rolling your own. By looking through the code below you may find one language appealing. Maybe it will reduce the anxiety about learning one of the languages.
test.py
import clr
import sys
sys.path.append('<path to dll>')
clr.AddReference("Microsoft.Rtc.Collaboration.dll")
print "Loaded ... "
from Microsoft.Rtc.Collaboration import *
from Microsoft.Rtc.Signaling import *
class Client:
def __init__(self,ownerUri,port):
self.__platform = None
self.__endpoint = None
self.__ownerUri = ownerUri
self.__port = port;
self.__conversation = None
self.__imCall = None
self.__imFlow = None
def get_Port(self):
return self.__port
def get_OwnerUri(self):
return self.__ownerUri
def Initialize(self,proxyPort):
settings = ServerPlatformSettings("UserAgent", "localhost", self.__port, self.__ownerUri + ";gruu")
self.__platform = CollaborationPlatform(settings)
self.__platform.EndStartup(self.__platform.BeginStartup(None, None))
settings = ApplicationEndpointSettings(self.__ownerUri,"localhost", proxyPort)
print "creating app endpoint"
self.__endpoint = ApplicationEndpoint(self.__platform, settings)
print 'Registering for incoming call'
self.__endpoint.RegisterForIncomingCall[InstantMessagingCall](self.__IMCallReceived)
print 'Establishing app endpoint'
self.__endpoint.EndEstablish(self.__endpoint.BeginEstablish(None,None))
def SendMessage(self,message):
self.__imFlow.EndSendInstantMessage(self.__imFlow.BeginSendInstantMessage(message,None,None))
def MakeCall(self,ownerUri):
self.__conversation = Conversation(self.__endpoint)
self.__imCall = InstantMessagingCall(self.__conversation)
self.__imCall.InstantMessagingFlowConfigurationRequested += self.__FlowConfigurationRequested
self.__imCall.StateChanged += self.__CallStateChanged
try:
self.__imCall.EndEstablish(self.__imCall.BeginEstablish(ownerUri, None, None, None, None))
except FailureResponseException,e:
print "Exception:", e.Message
def Terminate(self):
if self.__imCall != None:
self.__imCall.EndTerminate(self.__imCall.BeginTerminate(None,None))
def __CallStateChanged(self,sender,e):
s ="[" + self.__ownerUri + "] Call state changed: " + e.PreviousState.ToString() + " => " + e.State.ToString()
print s
def __IMCallReceived(self,sender,e):
print "Call received."
self.__imCall = e.Call
self.__imCall.InstantMessagingFlowConfigurationRequested += self.__FlowConfigurationRequested
self.__imCall.StateChanged += self.__CallStateChanged
e.Call.BeginAccept(self.__CallAcceptCompleted,None)
def __FlowConfigurationRequested(self,sender,e):
print "[",self.__ownerUri,"] Flow configuration requested."
self.__imFlow = e.Flow
e.Flow.StateChanged += self.__FlowStateChanged
e.Flow.MessageReceived += self.__MessageReceived
def __FlowStateChanged(self,sender,e):
s = "[" + self.__ownerUri + "] Flow state changed: " + e.PreviousState.ToString() + " => " + e.State.ToString()
print s
def __MessageReceived(self,sender,e):
s = '[' + self.__ownerUri + '] Message received: '+ e.TextBody
print s
def __CallAcceptCompleted(self,result):
try:
self.__imCall.EndAccept(result)
except RealTimeException,e:
print "Call accept failed:", e.Message
c1 = Client("sip:one@myhost", 5061)
c2 = Client("sip:two@myhost", 5062)
print 'Initializing client 1'
c1.Initialize(c2.get_Port())
print 'Initializing client 2'
c2.Initialize(c1.get_Port())
print 'Making call'
c1.MakeCall(c2.get_OwnerUri())
c1.SendMessage("Hello World!")
c1.Terminate()
c2.Terminate()
test.rb
require 'C:\windows\microsoft.net\assembly\GAC_MSIL\Microsoft.Rtc.Collaboration\<version>\Microsoft.Rtc.Collaboration.dll'
include Microsoft::Rtc::Collaboration
class Client
def initialize(ownerUri,port)
@platform = nil
@endpoint = nil
@ownerUri = ownerUri
@port = port
@conversation = nil
@imCall = nil
@imFlow = nil
end
def Port
@port
end
def OwnerUri
@ownerUri
end
def CallAcceptCompleted(result)
puts "Call accept completed."
@imCall.EndAccept(result)
#rescue Microsoft::Rtc::Signaling::RealTimeException => e
# puts "Accept failed: " + e.Message
#end
end
def IMCallReceived(sender,e)
puts 'Call received'
@imCall = e.Call
# The following should also be valid, but doesn't seem to work. Because of generics?
#
# @imCall.InstantMessagingFlowConfigurationRequested.add do |sender, e|
# FlowConfigurationRequested(sender,e)
# end
# In C#: call.InstantMessagingFlowConfigurationRequested += this.FlowConfigurationRequested
p = Proc.new { |sender,e| FlowConfigurationRequested(sender,e) }
@imCall.InstantMessagingFlowConfigurationRequested.add p
# In C#: call.StateChanged += this.CallStateChanged
p = Proc.new { |sender,e| CallStateChanged(sender,e) }
@imCall.StateChanged.add p
p = Proc.new { |result| CallAcceptCompleted(result) }
@imCall.BeginAccept(p,nil)
end
def MessageReceived(sender,e)
s = '[' + @ownerUri + '] Message received: ' + e.TextBody
puts s
end
def CallStateChanged(sender,e)
s = '[' + @ownerUri + '] Call state changed: ' + e.PreviousState.ToString() + '=> ' + e.State.ToString()
puts s
end
def FlowStateChanged(sender,e)
s = '[' + @ownerUri + '] Flow state changed: ' + e.PreviousState.ToString() + ' => ' + e.State.ToString()
puts s
end
def FlowConfigurationRequested(sender,e)
s = '[' + @ownerUri + '] Flow configuration requested'
puts s
@imFlow = e.Flow
p = Proc.new { |sender,e| FlowStateChanged(sender,e) }
@imFlow.StateChanged.add p
p = Proc.new { |sender,e| MessageReceived(sender,e) }
@imFlow.MessageReceived.add p
end
def Start(proxyPort)
settings = ServerPlatformSettings.new("UserAgent", "localhost", @port, @ownerUri + ";gruu")
@platform = CollaborationPlatform.new(settings)
puts 'Creating app endpoint'
settings = ApplicationEndpointSettings.new(@ownerUri, "localhost", proxyPort)
@endpoint = ApplicationEndpoint.new(@platform,settings)
# The C# equivalent of these two lines is:
#
# endpoint.RegisterForIncomingCall<InstantMessagingCall>(this.IMCallReceived)
#
p = Proc.new { |sender,e| IMCallReceived(sender,e) }
@endpoint.method(:RegisterForIncomingCall).of(InstantMessagingCall).call(p)
puts 'Creating platform'
@platform.EndStartup(@platform.BeginStartup(nil, nil))
@endpoint.EndEstablish(@endpoint.BeginEstablish(nil,nil))
end
def SendMessage(message)
@imFlow.EndSendInstantMessage(@imFlow.BeginSendInstantMessage(message,nil,nil))
end
def Terminate()
if @imCall then
puts '[' + @ownerUri + '] Terminating call.'
@imCall.EndTerminate(@imCall.BeginTerminate(nil,nil))
end
if @endpoint then
puts '[' + @ownerUri + '] Terminating endpoint.'
@endpoint.EndTerminate(@endpoint.BeginTerminate(nil,nil))
end
if @platform then
puts '[' + @ownerUri + '] Terminating platform.'
@platform.EndShutdown(@platform.BeginShutdown(nil,nil))
end
end
def MakeCall(ownerUri)
@conversation = Conversation.new(@endpoint)
@imCall = InstantMessagingCall.new(@conversation)
# In C#: call.InstantMessagingFlowConfigurationRequested += this.FlowConfigurationRequested
p = Proc.new { |sender,e| FlowConfigurationRequested(sender,e) }
@imCall.InstantMessagingFlowConfigurationRequested.add p
# In C#: call.StateChanged += this.CallStateChanged
p = Proc.new { |sender,e| CallStateChanged(sender,e) }
@imCall.StateChanged.add p
@imCall.EndEstablish(@imCall.BeginEstablish(ownerUri, nil, nil, nil, nil))
#rescue Microsoft::Rtc::Signaling::FailureResponseException => e
# print 'Exception: ' + e.Message
#end
end
end
c1 = Client.new("sip:one@myhost", 5061)
c2 = Client.new("sip:two@myhost", 5062)
puts 'Starting client 1'
c1.Start c2.Port
puts 'Starting client 2'
c2.Start c1.Port
puts 'Making call to ' + c2.OwnerUri
c1.MakeCall c2.OwnerUri
puts 'Sending message.'
c1.SendMessage("Hello world!")
puts 'Terminating client 1'
c1.Terminate
puts 'Terminating client 2'
c2.Terminate