Python Pandas MultiIndex and reading data from SQL Server

Python Pandas multiIndex is a hierarchical indexing over multiple tuples or arrays of data, enabling advanced dataframe wrangling and analysis on higher data dimensionality.

2019-06-26 05_49_58-Window

Everyone have had come across multiIndex in Python Pandas and had little annoyancens as the first time.

1. Python Pandas MultiIndex in SQL Server

With this in mind, we can create an example of pandas dataframe:

import pandas as pd

dt = pd.DataFrame([
                    ['2019-05-12','python',6],
                    ['2019-05-13','python',5],
                    ['2019-05-14','python',10],
                    ['2019-05-12','t-sql',12],
                    ['2019-05-13','t-sql',12],
                    ['2019-05-14','t-sql',12]
                  ],
                  columns = ['date','language','version'])

And the results are:

2019-06-05 17_46_50-Window

MultiIndex will give further data wrangling the capability of higher dimensionality wrangling. By simple:

EXEC sp_execute_external_script
	 @language = N'Python'
	,@script = N'
import pandas as pd

dt = pd.DataFrame([
                    [''2019-05-12'',''python'',6],
                    [''2019-05-13'',''python'',5],
                    [''2019-05-14'',''python'',10],
                    [''2019-05-12'',''t-sql'',12],
                    [''2019-05-13'',''t-sql'',12],
                    [''2019-05-14'',''t-sql'',12]
                  ],
                  columns = [''date'',''language'',''version''])

OutputDataSet = dt'
WITH RESULT SETS
((
  py_date SMALLDATETIME
 ,py_lang VARCHAR(10)
 ,py_ver TINYINT
))

So what happens, when we have data with primary key over multiple columns and we want to maintain this dimensionality in Python, as well as, propagate the results using Python Pandas with MultiIndex.

Now, let’s run the same example with additional option of adding pandas MultiIndex:

EXEC sp_execute_external_script
	 @language = N'Python'
	,@script = N'
import pandas as pd

dt = pd.DataFrame([
                    [''2019-05-12'',''python'',6],
                    [''2019-05-13'',''python'',5],
                    [''2019-05-14'',''python'',10],
                    [''2019-05-12'',''t-sql'',12],
                    [''2019-05-13'',''t-sql'',12],
                    [''2019-05-14'',''t-sql'',12]
                  ],
                  columns = [''date'',''language'',''version''])
dt.set_index([''language'', ''version''], inplace=True)
OutputDataSet = dt'
WITH RESULT SETS
((
 py_date SMALLDATETIME
,py_lang VARCHAR(10)
,py_ver TINYINT
))

We get in return an error message:

2019-06-26 06_06_14-Window

Msg 11537, Level 16, State 3, Line 135
EXECUTE statement failed because its WITH RESULT SETS clause specified 3 column(s) for result set number 1, but the statement sent 1 column(s) at run time.

This is due to the fact, how pandas MultiIndex operates. It in-places the index columns (hence multiIndex) into same column and keep it for higher dimensionality data wrangling. The error message would have not occurred, if I would have changed the RESULT SETS to a single column (py_date). Same can be seen in your favorite Python IDE, how columns are in-placed in index:

2019-06-26 06_12_00-Window

To make your exports using pandas dataframes in in-database Machine Learning services in your SQL Server, one way is to concatenate the index within Python, to preserve (and expose) the information of index.

-- MultiIndex with inplace True
-- adding a concatenated value
EXEC sp_execute_external_script
 @language = N'Python'
,@script = N'

import pandas as pd
dt = pd.DataFrame([
                    [''2019-05-12'',''python'',6],
                    [''2019-05-13'',''python'',5],
                    [''2019-05-14'',''python'',10],
                    [''2019-05-12'',''t-sql'',12],
                    [''2019-05-12'',''t-sql'',12],
                    [''2019-05-12'',''t-sql'',12]
                  ],
                  columns = [''date'',''language'',''version''])

dt[''PreservedIndex''] = dt[''language''].astype(str) + '';'' + \
 dt[''version''].astype(str)

dt.set_index([''language'', ''version''], inplace=True)
OutputDataSet=dt'
WITH RESULT SETS
((
  py_date SMALLDATETIME
 ,py_PreservedIndex VARCHAR(30)
))

And the results would be:

2019-06-26 06_24_40-Window

So we managed to exposed the constructed MultiIndex back to SQL Server.

2. How are SQL Server Indexes manages by Python Pandas

Continuing with the same example, let’s create a SQL Server table and with a Primary key constraint.

-- Going from T-SQL
-- And why T-SQL constraints does not play a role in Python

DROP TABLE IF EXISTS PyLang;
GO

CREATE TABLE PyLang (
	 [Date] DATETIME NOT NULL
	,[language] VARCHAR(10) NOT NULL
	,[version] INT
	,CONSTRAINT PK_PyLang PRIMARY KEY(date, language)
);
GO

INSERT INTO PyLang (date, language, version)
          SELECT '2019-05-12', 'python', 6
UNION ALL SELECT '2019-05-13', 'python', 5
UNION ALL SELECT '2019-05-14', 'python', 10
UNION ALL SELECT '2019-05-12', 't-sql', 12
UNION ALL SELECT '2019-05-13', 't-sql', 12
UNION ALL SELECT '2019-05-14', 't-sql', 12;

SELECT * FROM PyLang;
GO

with a simple dataset as:

2019-06-26 06_37_00-Window

With index created on Date and Language, we can easily override this in Python pandas and created different MultiIndex with different columns:

EXEC sp_execute_external_script
 @language = N'Python'
,@script = N'

import pandas as pd

dt = InputDataSet
dt[''PreservedIndex''] = dt[''language''].astype(str) + '';'' + \
 dt[''version''].astype(str)
dt.set_index([''language'', ''version''], inplace=True)
OutputDataSet=dt'
,@input_data_1 = N'SELECT * FROM PyLang'
WITH RESULT SETS
((
  py_date SMALLDATETIME
 ,py_MyPreservedIndex VARCHAR(40)
))

With the results

2019-06-26 06_41_08-Window

showing that  pandas MultiIndex was created using Language and Version column, as in T-SQL table, index was created on Date and Language.

Conclusion

  1. SQL Server and Python Pandas Indexes are two different worlds and should not be mixed.
  2. SQL Server uses Index primarily for DML operations and to keep data ACID.
  3. Python Pandas uses Index and MultiIndex for keeping data dimensionality when performing data wrangling and statistical analysis.
  4. SQL Server Index and Python Pandas Index don’t know about each other’s existence, meaning if user want to propagate the T-SQL index to Python Pandas (in order to minimize the impact of duplicates, missing values or to impose the relational model), it needs to be introduced and created, once data enters “in the python world”.
  5. When inserting Python Pandas MultiIndex results into SQL Server table with index, make sure that the data dimensionality fits together and in case of T-SQL primary keys, make sure, Python code is not producing any duplicates.
  6. Both, Python and T-SQL, indexes are help you having consistent data
  7. Performance is down-graded in both cases, if indexes are not used correctly or when overused.

 

As always, code is available at GitHub.

Happy coding.

Tagged with: , , , , , , ,
Posted in Uncategorized
One comment on “Python Pandas MultiIndex and reading data from SQL Server
  1. […] Tomaz Kastrun explains why you should never cross the streams: […]

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

Categories
Follow TomazTsql on WordPress.com
Programs I Use
Programs I Use
Programs I Use
Rdeči Noski – Charity

Rdeči noski

100% of donations made here go to charity, no deductions, no fees. For CLOWNDOCTORS - encouraging more joy and happiness to children staying in hospitals (http://www.rednoses.eu/red-noses-organisations/slovenia/)

€2.00

Top SQL Server Bloggers 2018
TomazTsql

Tomaz doing BI and DEV with SQL Server and R, Python and beyond

Discover

A daily selection of the best content published on WordPress, collected for you by humans who love to read.

Revolutions

Tomaz doing BI and DEV with SQL Server and R, Python and beyond

tenbulls.co.uk

attaining enlightenment with sql server, .net, biztalk, windows and linux

SQL DBA with A Beard

He's a SQL DBA and he has a beard

Reeves Smith's SQL & BI Blog

A blog about SQL Server and the Microsoft Business Intelligence stack with some random Non-Microsoft tools thrown in for good measure.

SQL Server

for Application Developers

Business Analytics 3.0

Data Driven Business Models

SQL Database Engine Blog

Tomaz doing BI and DEV with SQL Server and R, Python and beyond

Search Msdn

Tomaz doing BI and DEV with SQL Server and R, Python and beyond

R-bloggers

Tomaz doing BI and DEV with SQL Server and R, Python and beyond

Ms SQL Girl

Julie Koesmarno's Journey In Data, BI and SQL World

R-bloggers

R news and tutorials contributed by hundreds of R bloggers

Data Until I Die!

Data for Life :)

Paul Turley's SQL Server BI Blog

sharing my experiences with the Microsoft data platform, SQL Server BI, Data Modeling, SSAS Design, Power Pivot, Power BI, SSRS Advanced Design, Power BI, Dashboards & Visualization since 2009

Grant Fritchey

Intimidating Databases and Code

Madhivanan's SQL blog

A modern business theme

Alessandro Alpi's Blog

SQL Server, Azure and DLM in a nutshell :D

Paul te Braak

Business Intelligence Blog

Sql Server Insane Asylum (A Blog by Pat Wright)

Information about SQL Server from the Asylum.

Gareth's Blog

A blog about Life, SQL & Everything ...

SQLPam's Blog

Life changes fast and this is where I occasionally take time to ponder what I have learned and experienced. A lot of focus will be on SQL and the SQL community – but life varies.

%d bloggers like this: