Import Exposure via REST API Calls

In this section you will learn to use the Data Manipulation Tool, a tool based on Streamsets Data Collector, an open source low-latency ingest infrastructure application used to import, transfer, load and process data for later usage or storage in a database.

You will use a more complex scenario of importing exposure data from a Finastra open API.

To find out more about see the FusionCreator App Builder Documentation.

Get Started with Data Manipulation Tool

To get started with Data Manipulation Tool

1. In the Data source editor, click DATA MANIPULATION. The page opens and the list of the pipelines that you created is displayed.

Fig. 19: The Data Manipulation Tool page.

2. Click Create New Pipeline on the toolbar.
3. Enter the Title of the new pipeline: Get Exposure per Customer Rating
4. Click Save. The UI page switches to Edit mode.

Fig. 20: The Data Manipulation Tool page in pipeline editing mode.

5. On the Configuration section, go to Error Records tab and select Discard from the Error Records drop-down list. The pipeline error handling is now configured to discard the record. This will prevent the pipeline to break if something goes wrong while it is running. To find out more about pipeline error record handling, see the Data Manipulation Tool documentation.

To find out more about pipeline configuration, see the Data Manipulation Tool documentation.

Design a Pipeline for Data Import

For this section you must subscribe to Fusion Hello World Datasource product in the API Catalog. For details, see Get Started with Finastra Open APIs tutorial.

To retrieve the data with an origin stage

1. With the newly created Get Exposure per Customer Rating pipeline in Edit mode, filter out the Stage library, by selecting Origins, to display only the origin stages.
2. In the stage search box, start to type http client and select the HTTP Client stage from the filtered list.
3. In the Configuration section of the stage, click the HTTP tab.
4. Fill in the following fields:
   1. Resource URL: https://api.sandbox.fusionfabric.cloud/HelloData/V1/retail/data1
   2. Headers: click , and add the following request header:
      • Name: Ocp-Apim-Subscription-Key - here you provide the API subscription key; for details, see Get Started with Finastra Open APIs tutorial.
      • Value: copy the key from your API Developer Portal profile and and paste it here.
   3. Mode: Batch
5. Leave the defaults for the rest of the fields. For details about the values, point your mouse to , next to the field name.
6. In the Configuration section of the stage, click the Data format tab and modify the following parameter:
   • Max Object Length to 40960.

Fig. 21: The HTTP Client origin configuration.

To find out more about origin stages, see the documentation.

To process the data with a processor stage

1. Filter out the Stage library, by selecting Processors, to display only the processor stages.
2. In the stage search box, start to type field pivoter and select the Field Pivoter stage from the filtered list. The Field Pivoter processor stage is added to your pipeline.
3. Link the output of the HTTP Client origin to the input of the Field Pivoter processor.
4. In the Configuration section click the Field Pivot tab.
5. Fill in the following fields:
   1. Field to Pivot: /
   2. Copy All Fields: unselect this option.
6. Leave the defaults for the rest of the fields. For details about the values, point your mouse to , next to the field name.

Fig. 22: The Field Pivoter origin configuration.

You use the field pivoter to expand the nested JSON data structure that you received with the HTTP Client origin stage into a tabular data structure.

To find out more about processor stages, see the documentation.

To store the data using a destination stage

1. Filter out the Stage library, by selecting Destinations, to display only the destination stages.
2. In the stage search box, start to type cube insert and select the Cube Insert stage from the filtered list. The Cube Insert destination stage is added to your pipeline.
3. Link the output of the Field Pivoter processor to the input of the Cube Insert destination.
4. In the Cube Insert configuration tab click the Cube tab.
5. Fill in the following fields:
   1. Cube Name: Exposure. A new cube with this name will be created.
   2. Create report: select ALWAYS from the list. A report with the same name will be always created when you run this pipeline.
   3. Key Fields: fill in the following fields of the first key field:
      + Source field: /Id
   4. Value Fields: you have only one value field in your data. Enter the following details for the first value field:
      + Source Field: /Amount
      + Type: Double
6. Leave the defaults for the rest of the fields. For details about the values, point your mouse to , next to the field name.

Fig. 23: The Cube Insert destination configuration.

7. Run the pipeline. A confirmation message is displayed when finished.

Fig. 24: The confirmation message at the end of a successful pipeline run.

8. (Optional) In Data source editor go to CUBES and verify that the Exposure cube has been created. Click its name to see the columns. Also, go to REPORTS and verify that the Exposure report has been created, and click its name to see the attributes.

To find out more about destination stages, see the documentation.

Visualize the Data in a Chart

To display your data in a chart

1. Click to switch the UI to design mode.
2. Create a frame on the upper-right side of the board. This is the location where you will add a pie chart of your data.
3. On the Components bar, find and select the Pie Chart component.
4. Configure the chart as follows:
   1. Title: Exposure per Customer. Also click to show the chart title.
   2. Data source - search and select the Exposure report.
   3. Measures - select Amount.
   4. Dimensions - select the Customer column from the list of dimension columns.
   5. Chart type - select Doughnut.
   6. Legend - toggle the Visible option display the legend.
   7. Leave the defaults for the rest of the configuration. You can explore different settings for your chart.
5. Turn off the Design mode to visualize the bar chart on the Overview board.

Fig. 25: The pie chart component.