Learning Machine Learning using a Jet Engine

In this post I want to try and explain what machine learning is and put into context with what we used to do when analysing and manipulating data.  When I started doing all this the hot phrase was Business Intelligence (BI),  but this was preceded by EIS (Executive Information Systems) and DSS (Decision Support Systems).  All of these were to a larger extent looking backwards like driving a car using just the rear view mirror.  There was some work being done on trying to do look ahead (predictive analysis) and this was typically achieved by applying data mining techniques like regression (fitting points to a line or curve).

Nowadays the hot topic is Machine Learning (ML) so is that just a new marketing phrase or is this something that’s actually new?  Data mining and ML both have a lot in common; complex algorithms that have to be trained to build a model that can then be used against some set of data to derive something that wasn’t there before. This might be a numeric (continuous value) or a discreet value like the name of a group or a simple yes/no.  What I think makes ML different is that it’s there for a specific purpose where data mining is more like a fishing exercise to discover hidden relationships where one such use is predictive analytics.  So data mining could be considered to be a sub set of ML and one use of ML is to make predictions.

If I look at how Microsoft has implemented ML in Azure (which I will refer to as MAML), then there is a lot of processes around data acquisition before training and publishing occur.  In this regard we might relate this to a modern commercial jet engine.

• Suck the data in to the ML workbench..

This can be done either from a raw source or via HD Insight (Hadoop on Azure) which means MAML can work with big data.  Note that in the jet  engine diagram much of the air bypasses the engine if we are using big data in ML then we may ignore large chunks of that as not being relevant to what we are doing.  A good example is Twitter – most tweets aren’t relevant because they don’t mention my organisation.

• Squeeze the data.

If we haven’t sourced the data from something like HD Insight we need to clean it up to ensure  a high quality output and we understand it’s structure -  type, cardinality etc.

• Bang.

In a jet engine we introduce fuel and get a controlled explosion in ML we apply our analysis to get results and provide momentum for organisation.  Specifically this is where we build and refine our analytical model by taking an algorithm such as one used in data mining and training it against a sample set of data. MAML has specific features to support this a special split task for carving out data for model training and an evaluation task to tell you how well your model is performing by visualising the output against an ideal (upside down L curve).

• Blow. 

Having established that your model works you’ll want to unleash it on the world and share it which is analogous to the thrust or work produce by a jet engine.  In MAML we can expose the model as a web service or a data set which can be used in any number of ways to drive a decision making process. 

Jet engines don’t exist in isolation typically these are attached to aircraft and it is the aircraft that controls where the engine goes.  In ML we are going to have a project to set our direction and ML might only be a small part of this in the way that the product recommendation engine on a web site like Amazon is only a small part of the customer experience.  So as with all the varying names for projects that use data to drive businesses forward we need to be aware that the analytics bit is a small part of the whole; we need to be aware of data quality, customer culture and all of the baggage that are essential to a successful implementation.  There is also one extra complication that comes from the data mining world and that is that it’s not always possible to see how the result was derived at so we have to build trust in the results particularly if they contradict our intuition and experience. 

So that the good news for us experienced BI professionals is that we can apply many of the lessons we have learnt and apply them to ML and with MAML we don’t need to know too much about the data mining algorithms unless we want to.