Merging multiple model evaluation result in Azure ML

Evaluating prediction models in Azure ML  studio is always a must thing to do. But when your experiment consists of multiple models and you want to compare all at the same time, it might get a bit tricky.

Presume, we have a regression type of a problem and my experiment consists of three models.


After running the experiment, in order to check the scores of evaluation for each models, you need to check for each model separately.  And each time, the result is as following:


At this point we can use couple of ways to compare the models.

  1. By adding “Add rows” module two times one after another
  2. Merging results of two models into one “Evaluate model” module or
  3. Adding “Execute R script” module and “Add rows” module for cleaner output

By far the best is the third approach, because we manually add the labels into results knowing which model perform best.

When merging results of two models into one “Evaluate model” module, like shown on the diagram,


the main problem are the labels;


Intuitively, the logical order can be applied, but it does not mean, it will always be preserved. Moreover, when adding the the third model into the “Evaluate model” module, it can not be done – only two models can be compared. Therefore, “Add rows” module must be used.

So the experiment will look something like this:


Again the “Add Rows” module does not offer any ability to expose the model name and only some logical order can be applied when determining which result row correspond to which model.


In order to avoid confusion over labels of models (imagine having 20 models and you want to compare each one of them; and each of them is different not by the model name but by a particular parameter setting), we add “Execute R  script” module. Finalized experiment will look like:


With both modules “Execute R Script” and “Add Rows” creating the best final results with labels and models description.


Populating “Execute R Script” module with following R Code:

dataset1 <- maml.mapInputPort(1) # class: data.frame
data.set <- data.frame(Algorithm='Bayesian Linear Regression')
data.set <- cbind(data.set, dataset1[2:6])

Returned dataset can be build up to the data scientist.  It might also happen that not all algorithms return same metrics for evaluating the model. For example: “Bayesian Linear regression” model will return Negative Log likelihood where as “Linear regression” will not return this metrics. So when using R Script for output, please note that all merged datasets must have same number of columns with same names (more importantly, all consists of same metrics!).

After finally running the experiment, we get the following export:


And this is how the export of model comparison should be done. and now you don’t have to worry about mixing the names and models.


Posted in Uncategorized
2 comments on “Merging multiple model evaluation result in Azure ML
  1. Jacob says:

    Instead of creating and running 3 R scripts simply use Enter Data Manually in a CSV form, three times with the same heading “Algorithm” but different algorithm names below. The rest is the same.


  2. tomaztsql says:

    Thank you for the input.

    Best, Tomaž


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

Follow TomazTsql on

Tomaz doing BI and DEV with SQL Server and R

attaining enlightenment with sql server, .net, biztalk, windows and linux

SQL DBA with A Beard

He's a SQL DBA and he has a beard

DB NewsFeed

Matan Yungman's SQL Server blog

Reeves Smith's SQL & BI Blog

A blog about SQL Server and the Microsoft Business Intelligence stack with some random Non-Microsoft tools thrown in for good measure.

SQL Server

for Application Developers

Clocksmith Games

We make games we love to play

Business Analytics 3.0

Data Driven Business Models

SQL Database Engine Blog

Tomaz doing BI and DEV with SQL Server and R

Search Msdn

Tomaz doing BI and DEV with SQL Server and R


Tomaz doing BI and DEV with SQL Server and R

Ms SQL Girl

Julie Koesmarno's Journey In Data, BI and SQL World


R news and tutorials contributed by (750) R bloggers

Data Until I Die!

Data for Life :)

Paul Turley's SQL Server BI Blog

sharing my experiences with the Microsoft data platform, SQL Server BI, Data Modeling, SSAS Design, Power Pivot, Power BI, SSRS Advanced Design, Power BI, Dashboards & Visualization since 2009

Grant Fritchey

Intimidating Databases and Code

Madhivanan's SQL blog

A modern business theme

Alessandro Alpi's Blog

SQL Server, Azure and .net in a nutshell :D

Paul te Braak

Business Intelligence Blog

Sql Server Insane Asylum (A Blog by Pat Wright)

Information about SQL Server from the Asylum.

Gareth's Blog

A blog about Life, SQL & Everything ...

SQLPam's Blog

Life changes fast and this is where I occasionally take time to ponder what I have learned and experienced. A lot of focus will be on SQL and the SQL community – but life varies.

William Durkin

William Durkin a blog on SQL Server, Replication, Performance Tuning and whatever else.

$hell Your Experience !!!

As aventuras de um DBA usando o Poder do $hell

%d bloggers like this: