Feature/sfmc batchsource

README.md

@@ -3,7 +3,10 @@
 
 # Overview
 
-Contains CDAP pipeline plugins for Salesforce Marketing Cloud
+Contains CDAP pipeline plugins for Salesforce Marketing Cloud. Following plugins are available in this repository.
+
+* [Salesforce Marketing Cloud Source](docs/MarketingCloud-batchsource.md)
+* [Salesforce Marketing Cloud Data Extension Sink](docs/SalesforceDataExtension-batchsink.md)
 
 # Contact
 
@@ -14,7 +17,7 @@ CDAP User Group and Development Discussions:
 * [cdap-user@googlegroups.com](https://groups.google.com/d/forum/cdap-user)
 
 The *cdap-user* mailing list is primarily for users using the product to develop
-applications or building plugins for appplications. You can expect questions from
+applications or building plugins for applications. You can expect questions from
 users, release announcements, and any other discussions that we think will be helpful
 to the users.
 

docs/MarketingCloud-batchsource.md

@@ -0,0 +1,105 @@
+# Salesforce Marketing Cloud Source
+
+Description
+-----------
+
+Reads from one or multiple objects withing Salesforce Marketing Cloud depending on the mode values set for this plugin. 
+In case of `Multi Object` mode, the source will output a record for each row from the object it reads, with each record
+containing an additional field that holds the name of the object from which the record came from. In case of 
+`Single Object` mode, this additional field will not be there in the output. In addition, for each object that will be 
+read, this plugin will set pipeline arguments where the key is `multisink.[objectname]` and the value is the schema of 
+the object.
+
+Properties
+----------
+
+**Reference Name**: Name used to uniquely identify this source for lineage, annotating metadata, etc.
+
+**Data Retrieval Mode**: Mode of data retrieval. The mode can be one of two values: 
+
+`Multi Object` - will allow user to fetch data for multiple data extensions,  
+
+`Single Object` - will allow user to fetch data for single data extension.
+
+**Object**: Specify the object for which data to be fetched. This can be one of following values: 
+
+`Data Extension` - will allow user to fetch data for a single Data Extension object,
+
+`Email` - will allow user to fetch data for Email object,
+
+`Mailing List` - will allow user to fetch data for Mailing List object,
+
+`Bounce Event` - will allow user to fetch Tracking Bounce Events,
+
+`Open Event` - will allow user to fetch Tracking Open Events,
+
+`Unsub Event` - will allow user to fetch Tracking UnSubscribe Events,
+
+`Sent Event` - will allow user to fetch Tracking Sent Events,
+
+`UnSent Event` - will allow user to fetch Tracking UnSent Events.
+
+Note, this value will be ignored if the Mode is set to `Multi Object`.  
+
+**Data Extension External Key**: Specify the data extension key from which data to be fetched. Note, this value will 
+be ignored in following two cases: 
+
+* If the Mode is set to `Multi Object`
+* If the selected object name is other than `Data Extension`. 
+
+**Object List**: Specify the comma-separated list of objects for which data to be fetched; for example: 
+'Object1,Object2'. This can be one or more values from following possible values: 
+
+`Data Extension` - will allow user to fetch data for a single Data Extension object,
+
+`Email` - will allow user to fetch data for Email object,
+
+`Mailing List` - will allow user to fetch data for Mailing List object
+
+`Bounce Event` - will allow user to fetch Tracking Bounce Events,
+
+`Open Event` - will allow user to fetch Tracking Open Events,
+
+`Unsub Event` - will allow user to fetch Tracking UnSubscribe Events,
+
+`Sent Event` - will allow user to fetch Tracking Sent Events,
+
+`UnSent Event` - will allow user to fetch Tracking UnSent Events.
+
+Note, this value will be ignored if the Mode is set to `Single Object`.
+
+**Data Extension External Keys**: Specify the data extension keys from which data to be fetched; for example: 
+'Key1,Key2'. Note, this value will be ignored in following two cases: 
+
+* If the Mode is set to `Single Object`,
+
+* If the selected object list does not contain `Data Extension` as one of the objects.
+
+**Table Name Field**: The name of the field that holds the object name to which the data belongs to. Must not be the 
+name of any column for any of the objects that will be read. Defaults to `tablename`. In case of `Data Extension` 
+object, this field will have value in `dataextension_[Data Extension Key]` format. Note, the Table name field value 
+will be ignored if the Mode is set to `Single Object`.
+
+**Filter**: The filter selection criteria. eg: eventDate > '2010-01-01' and subscriberKey = 1001
+
+**Client ID**: OAuth2 client ID associated with an installed package in the Salesforce Marketing Cloud.
+
+**Client Secret**: TOAuth2 client secret associated with an installed package in the Salesforce Marketing Cloud.
+
+**Authentication Base URI**: Authentication Base URL associated for the Server-to-Server API integration. 
+For example, `https://instance.auth.marketingcloudapis.com/`
+
+**SOAP API Endpoint**: The SOAP Endpoint URL associated for the Server-to-Server API integration. For example, 
+`https://instance.soap.marketingcloudapis.com/Service.asmx`
+
+**REST API Base URI**: The REST API Base URL associated for the Server-to-Server API integration. For example, 
+`https://instance.rest.marketingcloudapis.com/`
+
+**Error handling:** Whether to fail the pipeline if an error encountered while reading records. This can be one of 
+following values:
+
+`Stop on error` - will fail the pipeline if an error encountered,  
+
+`Skip on error` - will not fail the pipeline even if an error encountered and will continue the pipeline.
+
+Note: the default value is `Skip on error`.

examples/example_sfmc_multiobject_pipeline-cdap-data-pipeline.json

@@ -0,0 +1,101 @@
+{
+    "artifact": {
+        "name": "cdap-data-pipeline",
+        "version": "6.2.1",
+        "scope": "SYSTEM"
+    },
+    "description": "Data Pipeline Application",
+    "name": "example_sfmc_multiobject_pipeline",
+    "config": {
+        "resources": {
+            "memoryMB": 2048,
+            "virtualCores": 1
+        },
+        "driverResources": {
+            "memoryMB": 2048,
+            "virtualCores": 1
+        },
+        "connections": [
+            {
+                "from": "Salesforce-Marketing",
+                "to": "File"
+            }
+        ],
+        "comments": [],
+        "postActions": [],
+        "properties": {},
+        "processTimingEnabled": true,
+        "stageLoggingEnabled": false,
+        "stages": [
+            {
+                "name": "Salesforce-Marketing",
+                "plugin": {
+                    "name": "MarketingCloud",
+                    "type": "batchsource",
+                    "label": "Salesforce Marketing",
+                    "artifact": {
+                        "name": "salesforce-marketing-plugins",
+                        "version": "1.2.2-SNAPSHOT",
+                        "scope": "USER"
+                    },
+                    "properties": {
+                        "queryMode": "Multi Object",
+                        "objectList": "Data Extension",
+                        "tableNameField": "tablename",
+                        "failOnError": "false",
+                        "referenceName": "SFMC",
+                        "clientId": "${SFMC_CLIENT_ID}",
+                        "clientSecret": "${SFMC_CLIENT_SECRET}",
+                        "authEndpoint": "${SFMC_AUTH_URI}",
+                        "soapEndpoint": "${SFMC_SOAP_URI}",
+                        "restEndpoint": "${SFMC_REST_URI}",
+                        "dataExtensionKeyList": "CA397115-F3E1-4180-B7D0-AF91A60E8B2D,CFA3EE8B-320E-4E27-BBCC-CF4291556BE0"
+                    }
+                },
+                "outputSchema": [
+                    {
+                        "name": "etlSchemaBody",
+                        "schema": ""
+                    }
+                ]
+            },
+            {
+                "name": "File",
+                "plugin": {
+                    "name": "File",
+                    "type": "batchsink",
+                    "label": "File",
+                    "artifact": {
+                        "name": "core-plugins",
+                        "version": "2.4.1",
+                        "scope": "SYSTEM"
+                    },
+                    "properties": {
+                        "suffix": "yyyy-MM-dd-HH-mm",
+                        "format": "csv",
+                        "referenceName": "SFMCData",
+                        "path": "${OUTPUT_DIR}",
+                        "schema": "{\"type\":\"record\",\"name\":\"etlSchemaBody\",\"fields\":[{\"name\":\"subscriber_key\",\"type\":[\"string\",\"null\"]},{\"name\":\"campaign_c\",\"type\":[\"string\",\"null\"]},{\"name\":\"email_address\",\"type\":[\"string\",\"null\"]},{\"name\":\"campaign_a\",\"type\":[\"string\",\"null\"]},{\"name\":\"account_unsub\",\"type\":[\"string\",\"null\"]},{\"name\":\"campaign_b\",\"type\":[\"string\",\"null\"]},{\"name\":\"first__name\",\"type\":[\"string\",\"null\"]},{\"name\":\"emailaddress\",\"type\":[\"string\",\"null\"]},{\"name\":\"subscriberkey\",\"type\":[\"string\",\"null\"]},{\"name\":\"tablename\",\"type\":\"string\"}]}"
+                    }
+                },
+                "outputSchema": [
+                    {
+                        "name": "etlSchemaBody",
+                        "schema": "{\"type\":\"record\",\"name\":\"etlSchemaBody\",\"fields\":[{\"name\":\"subscriber_key\",\"type\":[\"string\",\"null\"]},{\"name\":\"campaign_c\",\"type\":[\"string\",\"null\"]},{\"name\":\"email_address\",\"type\":[\"string\",\"null\"]},{\"name\":\"campaign_a\",\"type\":[\"string\",\"null\"]},{\"name\":\"account_unsub\",\"type\":[\"string\",\"null\"]},{\"name\":\"campaign_b\",\"type\":[\"string\",\"null\"]},{\"name\":\"first__name\",\"type\":[\"string\",\"null\"]},{\"name\":\"emailaddress\",\"type\":[\"string\",\"null\"]},{\"name\":\"subscriberkey\",\"type\":[\"string\",\"null\"]},{\"name\":\"tablename\",\"type\":\"string\"}]}"
+                    }
+                ],
+                "inputSchema": [
+                    {
+                        "name": "Salesforce Marketing",
+                        "schema": ""
+                    }
+                ]
+            }
+        ],
+        "schedule": "0 * * * *",
+        "engine": "spark",
+        "numOfRecordsPreview": 100,
+        "description": "Data Pipeline Application",
+        "maxConcurrentRuns": 1
+    }
+}

examples/example_sfmc_openevent_withfilter-cdap-data-pipeline.json

@@ -0,0 +1,94 @@
+{
+    "artifact": {
+        "name": "cdap-data-pipeline",
+        "version": "6.2.1",
+        "scope": "SYSTEM"
+    },
+    "description": "",
+    "name": "example_sfmc_openevent_withfilter",
+    "config": {
+        "resources": {
+            "memoryMB": 2048,
+            "virtualCores": 1
+        },
+        "driverResources": {
+            "memoryMB": 2048,
+            "virtualCores": 1
+        },
+        "connections": [
+            {
+                "from": "Salesforce-Marketing",
+                "to": "File"
+            }
+        ],
+        "comments": [],
+        "postActions": [],
+        "properties": {},
+        "processTimingEnabled": true,
+        "stageLoggingEnabled": false,
+        "stages": [
+            {
+                "name": "Salesforce-Marketing",
+                "plugin": {
+                    "name": "MarketingCloud",
+                    "type": "batchsource",
+                    "label": "Salesforce Marketing",
+                    "artifact": {
+                        "name": "salesforce-marketing-plugins",
+                        "version": "1.2.2-SNAPSHOT",
+                        "scope": "USER"
+                    },
+                    "properties": {
+                        "queryMode": "Single Object",
+                        "objectName": "Open Event",
+                        "failOnError": "false",
+                        "referenceName": "SFMC",
+                        "clientId": "${SFMC_CLIENT_ID}",
+                        "clientSecret": "${SFMC_CLIENT_SECRET}",
+                        "authEndpoint": "${SFMC_AUTH_URI}",
+                        "soapEndpoint": "${SFMC_SOAP_URI}",
+                        "restEndpoint": "${SFMC_REST_URI}",
+                        "filter": "subscriberKey = 'manoj.muralidharan@verticurl.com' and eventDate > '04-19-2019'"
+                    }
+                },
+                "outputSchema": [
+                    {
+                        "name": "etlSchemaBody",
+                        "schema": "{\"type\":\"record\",\"name\":\"etlSchemaBody\",\"fields\":[{\"name\":\"id\",\"type\":[\"string\",\"null\"]},{\"name\":\"sendID\",\"type\":[\"int\",\"null\"]},{\"name\":\"subscriberKey\",\"type\":[\"string\",\"null\"]},{\"name\":\"eventDate\",\"type\":[\"string\",\"null\"]},{\"name\":\"eventType\",\"type\":[\"string\",\"null\"]},{\"name\":\"triggeredSendDefinitionObjectID\",\"type\":[\"string\",\"null\"]},{\"name\":\"batchID\",\"type\":[\"int\",\"null\"]}]}"
+                    }
+                ]
+            },
+            {
+                "name": "File",
+                "plugin": {
+                    "name": "File",
+                    "type": "batchsink",
+                    "label": "File",
+                    "artifact": {
+                        "name": "core-plugins",
+                        "version": "2.4.1",
+                        "scope": "SYSTEM"
+                    },
+                    "properties": {
+                        "suffix": "yyyy-MM-dd-HH-mm",
+                        "format": "csv",
+                        "schema": "{\"type\":\"record\",\"name\":\"etlSchemaBody\",\"fields\":[{\"name\":\"id\",\"type\":[\"string\",\"null\"]},{\"name\":\"sendID\",\"type\":[\"int\",\"null\"]},{\"name\":\"subscriberKey\",\"type\":[\"string\",\"null\"]},{\"name\":\"eventDate\",\"type\":[\"string\",\"null\"]},{\"name\":\"eventType\",\"type\":[\"string\",\"null\"]},{\"name\":\"triggeredSendDefinitionObjectID\",\"type\":[\"string\",\"null\"]},{\"name\":\"batchID\",\"type\":[\"int\",\"null\"]}]}",
+                        "referenceName": "FILEOUT",
+                        "path": "${OUTPUT_DIR}"
+                    }
+                },
+                "outputSchema": "{\"type\":\"record\",\"name\":\"etlSchemaBody\",\"fields\":[{\"name\":\"id\",\"type\":[\"string\",\"null\"]},{\"name\":\"sendID\",\"type\":[\"int\",\"null\"]},{\"name\":\"subscriberKey\",\"type\":[\"string\",\"null\"]},{\"name\":\"eventDate\",\"type\":[\"string\",\"null\"]},{\"name\":\"eventType\",\"type\":[\"string\",\"null\"]},{\"name\":\"triggeredSendDefinitionObjectID\",\"type\":[\"string\",\"null\"]},{\"name\":\"batchID\",\"type\":[\"int\",\"null\"]}]}",
+                "inputSchema": [
+                    {
+                        "name": "Salesforce Marketing",
+                        "schema": "{\"type\":\"record\",\"name\":\"etlSchemaBody\",\"fields\":[{\"name\":\"id\",\"type\":[\"string\",\"null\"]},{\"name\":\"sendID\",\"type\":[\"int\",\"null\"]},{\"name\":\"subscriberKey\",\"type\":[\"string\",\"null\"]},{\"name\":\"eventDate\",\"type\":[\"string\",\"null\"]},{\"name\":\"eventType\",\"type\":[\"string\",\"null\"]},{\"name\":\"triggeredSendDefinitionObjectID\",\"type\":[\"string\",\"null\"]},{\"name\":\"batchID\",\"type\":[\"int\",\"null\"]}]}"
+                    }
+                ]
+            }
+        ],
+        "schedule": "0 * * * *",
+        "engine": "spark",
+        "numOfRecordsPreview": 100,
+        "maxConcurrentRuns": 1
+    }
+}