Data Factory: AzureSQL in- and output for pipeline activity type AzureMLBatchExecution

Question

In Azure Data Factory, I’m trying to call an Azure Machine Learning model by a Data Factory Pipeline. I want to use an Azure SQL table as input and another Azure SQL table for the output. First I deployed a Machine Learning (classic) web service. Then I created an Azure Data Factory Pipeline, using a LinkedService (type= ‘AzureML’, using Request URI and API key of the ML-web service) and an input and output dataset (‘AzureSqlTable’ type).

Deploying and Provisioning is succeeded. The pipeline starts as scheduled, but keeps ‘Running’ without any result. The pipeline activity is not being shown in the Monitor&Manage: Activity Windows.

On different sites and tutorials, I only find JSON-scripts using the activity type ‘AzureMLBatchExecution’ with BLOB in- and outputs. I want to use AzureSQL in- and output but I can’t get this working.

Can someone provide a sample JSON-script or tell me what’s possibly wrong with the code below?

{

    "name": "Predictive_ML_Pipeline",

    "properties": {

        "description": "use MyAzureML model",

        "activities": [

            {

                "type": "AzureMLBatchExecution",

                "typeProperties": {},

                "inputs": [

                    {

                        "name": "AzureSQLDataset_ML_Input"

                    }

                ],

                "outputs": [

                    {

                        "name": "AzureSQLDataset_ML_Output"

                    }

                ],

                "policy": {

                    "timeout": "02:00:00",

                    "concurrency": 3,

                    "executionPriorityOrder": "NewestFirst",

                    "retry": 1

                },

                "scheduler": {

                    "frequency": "Week",

                    "interval": 1

                },

                "name": "My_ML_Activity",

                "description": "prediction analysis on ML batch input",

                "linkedServiceName": "AzureMLLinkedService"

            }

        ],

        "start": "2017-04-04T09:00:00Z",

        "end": "2017-04-04T18:00:00Z",

        "isPaused": false,

        "hubName": "myml_hub",

        "pipelineMode": "Scheduled"

    }

}

Answer 1

The input file of Azure SQL is as follows:

{

"name": "AzureSQLDataset_ML_Input",

"properties": {

    "published": false,

    "type": "AzureSqlTable",

    "linkedServiceName": "SRC_SQL_Azure",

    "typeProperties": {

        "tableName": "dbo.Azure_ML_Input"

    },

    "availability": {

        "frequency": "Week",

        "interval": 1

    },

    "external": true,

    "policy": {

        "externalData": {

            "retryInterval": "00:01:00",

            "retryTimeout": "00:10:00",

            "maximumRetry": 3

        }

    }

}