Hi All,

The goal of this document is to provide field level details for Debmas idoc, used to load and extract data of Customer Master in SAP.

In our daily job, often we come across a scenario where we need to load data into SAP or extract data from SAP for Customer/Employee/Vendor/Material etc.

We understand that data in SAP are stored in SAP tables, with specific field requirement like data type and Length of field.

Often it’s difficult to memorize the field level details for any table.

In my sincere attempt, I have tried to include the field level details for a particular case of Customer Master.

In Customer Master, we can load the data in, as well as extract from, SAP, using DEBMAS idoc.

Please find attached the document, to understand the meaning of each and every field in IDOC, its system requirement (mandatory/non Mandatory), field name, description, segment name and table name where the field lies in SAP.

The document is designed segment wise.

Hope this would be helpful for all.

Thanks,

Mayank Mehta

A proper data reconciliation process must be in place in any data Extraction-Transformation-Load (ETL) process. A successful reconciliation process should only indicate whether or not the data is correct. But data reconciliation is not easy. Fortunately for us, Data Services (BODS) provides an in-built data reconciliation feature called Auditing. Auditing is a way to ensure that a dataflow loads correct data into the target warehouse. Let's see how...

Audit objects are used to collect run time audit statistics about the objects within a data flow. Using the Audit feature we can collect statistics about data read into a job, processed by various transforms, and loaded into targets.

We define audit rules to determine if the correct data is processed. Also in the event of audit rule failure there is a provision to generate notification of audit failures.

Auditing stores these statistics in the repository for further future analysis. In event of any mismatch in source/target row counts or sum of measures notification can be set, to highlight the audit failure for our immediate action like reload, identify incorrect source data etc.

Now let us see how easy it is to enable Audit option in Data Services. Find below a sample dataflow on which we will Audit the count of Source and Target record count and also will verify the HASH value of character data type columns between Source and Target.

We can enable the Audit option property of the dataflow either by right-click the Dataflow in Workspace or from the Local Object Library.  

On selection of Audit, we have two tabs namely Label And Rule which is described in details below.

Auditing Features

Following are the various available features of auditing:

§  Define Audit Points to collect run-time statistics about the data. E.g. Number of rows extracted from Source, Processed by Transforms, Loaded into Targets

§  Define Audit Rules for the audit statistics to ensure expected data at the audit points in the dataflow.

§  Generate run-time notification for the audit rule that fails i.e. Action on failure and the values of the audit statistics for which the rule failed.

§  Display Audit Statistics after the job execution to identify the object/transform that processed incorrect data in the dataflow.

Auditing Objects in a Dataflow

§  Audit Point– This is the object in a dataflow where audit statistics is collected. i.e. Source, Transform or Target.

§  Audit Function– Defines the collects audit statistics for a table, output schema or column. E.g. Count, Sum, Average, Checksum

§  Audit Label– This is a unique name generated by BODS for each audit function that is defined on an audit point to collect the audit statistics.

§  Audit Rule– A Boolean expression which uses audit labels to verify a job.

§  Action on Audit Failure– Ways to generate notification for audit rules failure.

Audit Labels

The software generates a unique name for each audit function that we define on an audit point and are editable.

If the audit point is on a table or output schema for the audit function Count, the software generates two labels namely $Count_objectname and $CountError_objectname

If the audit point is on a column, the software generates an audit label namely $auditfunction_objectname

If the audit point is in an embedded data flow, the labels generated are namely $Count_objectname_embeddedDFname, $CountError_objectname_embeddedDFname and $auditfunction_objectname_embeddedDFname

Audit Functions

There are four Audit Functions available in BODS.

§  COUNT function helps to get the audit information on a table or output schema. The default datatype is INTEGER.

§  SUM function helps to generate audit information like summation of measure type columns. E.g. Revenue. Accepted datatypes are INTEGER, DECIMAL, DOUBLE, REAL.

§  AVERAGE function helps to generate audit information like average of measure type columns. E.g. Profit Margin. Accepted datatypes are INTEGER, DECIMAL, DOUBLE, REAL.

§  CHECKSUM function helps to audit based on the hash values generated for VARCHAR datatype columns. The order of rows is important for the result of CHECKSUM function.

Error Count Statistics

BODS collects two types of statistics when we use a Count audit function. Good row count for rows processed without any error. Error row count for rows that the job could not process but ignores those rows to continue processing. When we specify the Use overflow file option in the Source or Target Editor we can handle the error rows.

Let us now see how to define the Audit Points and the Audit Functions. We can also modify the default Audit Labels generated by Data Services if necessary.

Audit Rule

An Audit Rule is a Boolean expression which consists of a Left-Hand-Side (LHS), a Boolean operator, and a Right-Hand-Side (RHS). The LHS can be a single audit or multiple audit labels that form an expression with one or more mathematical operators, or a function with audit labels as parameters. The RHS can also be a single or multiple audit labels that form an expression with one or more mathematical operators, a function with audit labels as parameters, or a constant. Examples:

The following Boolean expressions are examples of audit rules:

§  $Count_ODS_CUSTOMER = $Count_DW_CUSTOMER

§  $Sum_CALLS_PREPAID + $Sum_CALLS_POSTPAID = $Sum_DW_CALLS

§  round($Avg_CALL_TIME) >= 30

Audit Notification

There are three types of Actions available for Notification in the event of an Audit Failure namely Email to list, Script and Raise exception. We can select either all three actions or can choose any combination of the actions. Find below the order these actions are executed in the event when all the actions are selected as a part of notification for audit failure.

§  Email to list— BODS sends a notification of which audit rule failed to the email addresses mentioned for this option.

§  Script— BODS executes the custom script mentioned for this option in the event of Audit rule failure.

§  Raise exception— The job fails if an Audit rule fails, and the error log shows which audit rule failed. The job stops at the first audit rule that fails. This action is the default. We can continue with the job execution if we place the audit exception in a try/catch block.

We need to the define the Audit Rules related to the Audit Points or Audit Labels. Based on the success or failure of this rule we will generate error message.

Viewing Audit Results

We can find the audit status in the Job Monitor Log, when we set Audit Trace to Yes on the Trace tab in the Execution Properties window of the Job. We can see messages for audit rules that passed and failed. If the audit rule fails, the places that display audit information depends on the Action on failure option that was selected.

§  When Raise exception action is selected and an audit rule fails the Job Error Log and the Metadata Reports shows the rule that failed.

§  In case of Email to list we get notified of audit rule failure via Email message. Audit rule statistics are also available in the Metadata Reports.

§  For failure action type Script the custom script is executed and the audit statistics can be viewd from the Metadata Reports.

Metadata Reports

We can view passed and failed audit rules result in the metadata reports. We can look at the Audit Status column in the Data Flow Execution Statistics reports of the Metadata Report tool. This Audit Status column has the following values:

§  Not Audited

§  Passed— All audit rules succeeded. This value is a link to the Auditing Details report which shows the audit rules and values of the audit labels.

§  Information Collected— This status occurs when we define audit labels to collect statistics but we do not define audit rules. This value is a link to the Auditing Details report which shows the values of the audit labels.

§  Failed— Audit rule failed. This value is a link to the Auditing Details report which shows the rule that failed and values of the audit labels.

Notes

• 1.    An audit label can become invalid if we delete the audit label in an embedded data flow that the parent data flow has enabled.
• 2.    An audit label can become invalid if we delete or rename an object that had an audit point defined on it.
• 3.    We can edit the audit label name while creating the audit function and before creating an audit rule that uses the label.
• 4.    If we edit the label name after using it in an audit rule, the audit rule does not automatically use the new name. We must redefine the rule with the new label name.
• 5.    If we define multiple rules in a data flow, all rules must succeed or the audit fails.
• 6.    Auditing is disabled when you run a job with the debugger.
• 7.    We cannot audit NRDM schemas or real-time jobs.
• 8.    We cannot audit within an ABAP Dataflow, but we can audit the output of an ABAP Dataflow.
• 9.    If we use the CHECKSUM audit function in a job that normally executes in parallel, the software disables the DOP for the whole data flow. The order of rows is important for the result of CHECKSUM, and DOP processes the rows in a different order than in the source.
• 10.  We can audit the number of rows loaded using bulkload dataflow only for those that uses the Oracle API method.
• 11.  While executing the job, the Enable auditing option in the Execution Properties window is checked by default. If we do not want to collect audit statistics for the specific job execution clear the selection.
• 12.  If we clear the action Raise exception and an audit rule fails, the job will complete successfully and the audit does not write messages to the job log. We can view which rule failed in the Auditing Details report in the Metadata Reporting tool.
• 13.  If a pushdown_sql function is after an audit point, the software cannot execute it.
• 14.  If we add an audit point prior to an operation that is usually pushed down to the database server, performance might degrade because pushdown operations cannot occur after an audit point.

When we audit the output data of an object, the optimizer cannot pushdown operations after the audit point. Therefore, if the performance of a query that is pushed to the database server is more important than gathering audit statistics from the source, we should define the first audit point on the query or later in the data flow. For example, suppose a query has a WHERE clause that is pushed to the database server that significantly reduces the amount of data that returns to the software. So we can define the first audit point on the query, rather than on the source, to obtain audit statistics on the query results.

Audit Result Logging

In case we want to capture or log the Audit Results to Database tables we can use the custom script as:

$Count_ODS_CUSTOMER = $Count_CUST_DIM

AND sql('Target_DS', 'insert into dbo.STG_RECON values (

\'ODS_CUSTOMER\',[ $Count_ODS_CUSTOMERM ],

\'CUST_DIM\',[ $Count_CUST_DIM ],

{ to_char( sysdate(), \'YYYY-MM-DD\' ) } )

') is NULL

Auto Correct Load:

Auto correct loading ensures that the same row is not duplicated in a target table

Input Data Screen Short:

Target Option Window: Following steps need to be done

Uncheck the Delete sata from Table before Loading

Before Updating in Update Control:

After Updating:

Now go to Source Data and add two Duplicate rows

Before Updating:

After Updating:

After Creation of Jobs in Data Services Designer

Now Apply Auto Correct Load on Target Tables

Note: After apply Auto Correct Load Duplicate values are removed from Target Tables

Now Execute the Job

Output:

Ignore Column Option:

If a matching row exists, the row is updated depending on the values of Ignore columns with value, and Ignore columns with null:

Ignore Columns Options are Worked with Auto Correct Load:

• When the Ignore columns with null option is set to Yes and the column data from the source is NULL, then the corresponding column in the target table is not updated. Otherwise, the corresponding target column is updated as NULL since the source column is NULL.

Now Go To Target Table Properties:

Before Update in the Target Table Properties:

After Update in the Target Table Properties:

Now Source Data Information:

Now create Jobs in Data Services Using Job-Work Flow->Data Flow-> Source & Target and make EMP_ID as a Primary key

View of Source table Before Loading:

View of Source table After Null Values Before Loading:

Now update in the Target Table before loading the data

Then Finally Execute the Job

Output after executing the job:

Ignore Column with Value Option:

If a matching row exists, the row is updated depending on the values of Ignore columns with value, and Ignore columns with null:

• When the column data from the source matches the value in Ignore columns with value, the corresponding column in the target table is not updated. The value may be spaces. Otherwise, the corresponding column in the target is updated with the source data.

Now Go To Target Table Properties:

Before Update in the Target Table Properties:

Before Update in the Target Table Properties:

After Update in the Target Table Properties:

Now Source Data Information:

Now create Jobs in Data Services Using Job-Work Flow->Data Flow-> Source & Target and make EMP_ID as a Primary key

View of Source table Before Loading:

Now update in the Target Table before loading the data

Then Finally Execute the Job

Output after executing the job:

Update Key Columns in Update Control:

Now Go To Target Table Properties:

Before Update in the Target Table Properties:

After Update in the Target Table Properties:

Now Source Data Information:

Before Update:

After Update in Source Table:

Now create Jobs in Data Services Using Job-Work Flow->Data Flow-> Source & Target and make EMP_ID as a Primary key

View of Source table Before Loading:

Now update in the Target Table before loading the data

Then Finally Execute the Job

Output after executing the job:

What is dirty data costing you?

Intrinsically, you know that you need good data. But how far do you need to go? What are the real costs incurred if you DON’T have clean data? Check out this new SAP Data Quality infographic, which highlights the probably extent of your data quality problems, and the costs associated with those problems. This blog by Ina Felsheim also links to the popular Information Governance infographic.

Data Profiling and Data Cleansing – Use Cases and Solutions at SAP

While some organizations have set up enterprise wide data governance projects including managing and tightly integrating People, Processes, Policies & Standards, Metrics and Tools other companies are still in the starting phase of mostly departmental data cleansing activities. Recent research from Gartner still indicate that the poor data quality is a primary reason for about 40% of all business initiatives fail. Read more in this blog by Niels Weigel.

Data Quality Performance Guide

This document from Dan Bills provides performance throughput numbers for data quality capabilities within Data Quality Management and Data Services v4.2.

Hallo,

I have problems running a Job with ODBC MySQL Datastore Sourcetable.

The job get an error message CON-120302 when trying to connect the MySQL Datastore Source Table:

My SLES 11 production with new Data Services 4.1 shows errors when I try to use a MySQL Datasource.

SAP BusinessObjects Data Services

   Version: 14.1.1.392

Linux 3.0.74-0.6.8.1.5506.2.PTF-default #1 SMP Wed May 15 07:26:33 UTC 2013 (5e244d7) x86_64 x86_64 x86_64 GNU/Linux

Old driver from running Data Services 3.2 in SLES 10 was libmyodbc3-3.51.27.so .

Now in SLES 11 it states:

ldd /usd/as16148a/soft/mysql-connector-odbc/usr/lib64/libmyodbc3-3.51.27.so

        linux-vdso.so.1 =>  (0x00007ffff7ffe000)

        libdl.so.2 => /lib64/libdl.so.2 (0x00007ffff7ace000)

        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007ffff78b1000)

        libltdl.so.3 => not found

        libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007ffff7675000)

        libnsl.so.1 => /lib64/libnsl.so.1 (0x00007ffff745d000)

        libm.so.6 => /lib64/libm.so.6 (0x00007ffff71e4000)

        libodbcinst.so.1 =>

/usd/as16148a/soft/unixODBC/usr/local/lib/libodbcinst.so.1

(0x00007ffff70cd000)

        libc.so.6 => /lib64/libc.so.6 (0x00007ffff6d56000)

        /lib64/ld-linux-x86-64.so.2 (0x0000555555554000)

libltdl.so.3 => not found

So i try new mysql odbc driver libmyodbc5w.so

isql 'MySQL S-Web' s-web-reader ###

+---------------------------------------+

| Connected!                            |

|                                       |

| sql-statement                         |

| help [tablename]                      |

| quit                                  |

|                                       |

+---------------------------------------+

SQL> quit

odbcinst -j

unixODBC 2.2.14

DRIVERS............:

/usd/as16148a/home/serberw/businessobjects/dataservices/bin/odbcinst.iniSYSTEM DATA SOURCES:

/usd/as16148a/home/serberw/businessobjects/dataservices/bin/odbc.ini

FILE DATA SOURCES..:

/usd/as16148a/home/serberw/businessobjects/dataservices/bin/ODBCDataSources

USER DATA SOURCES..:

/usd/as16148a/home/serberw/businessobjects/dataservices/bin/odbc.ini

SQLULEN Size.......: 8

SQLLEN Size........: 8

SQLSETPOSIROW Size.: 8

ldd /usd/as16148a/soft/mysql-connector-odbc/usr/lib64/libmyodbc5w.so | less

        linux-vdso.so.1 =>  (0x00007ffff7ffe000)

        libodbc.so.1 => /usd/as16148a/soft/unixODBC/usr/local/lib/libodbc.so.1 (0x00007ffff787b000)

        librt.so.1 => /lib64/librt.so.1 (0x00007ffff7664000)

        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007ffff7447000)

        libodbcinst.so.1 => /usd/as16148a/soft/unixODBC/usr/local/lib/libodbcinst.so.1 (0x00007ffff7330000)

        libdl.so.2 => /lib64/libdl.so.2 (0x00007ffff712c000)

        libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00007ffff6e22000)

        libm.so.6 => /lib64/libm.so.6 (0x00007ffff6ba8000)

        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007ffff6992000)

        libc.so.6 => /lib64/libc.so.6 (0x00007ffff661b000)

        /lib64/ld-linux-x86-64.so.2 (0x0000555555554000)

lines 1-11/11 (END)

DSConnectionManager.sh -c does not work.

The job get an error message CON-120302 when trying to connect the MySQL Datastore Source Table:

CON-120302: |Session J_3150_AWZT_SWEB- ODBC-Aufruf <SQLConnect> für Datenquelle <MySQL S-Web> fehlgeschlagen: <瑠ɱ>. Benachrichtigen Sie den Kundensupport.

My SLES 11 production use a technical user for running and there are no root privileges allowed neither for installing nor for running the production of Data Services 4.1.

Are there any helpfull experience with SLES 11, Data Services 4.1 and MySQL ODBC.

Thank you,

Rolf-Dieter

Attached document will help to understand how to Create, Configure and Manage Central Repository (CR) in BODS 4.0.

Note: Open each slide to get better vision.

Thanks,

Sagar Girme

Purpose:

The purpose of this document is to re-install the cleansing package in SAP Business Objects Data Services 4.X

Overview:

If you forget to deploy "cleansing package in SAP Business Objects Data Services 4.X" then please refer below points to activate it again, Below screen shots is of Windows Environment.

Path in My Virtual Machine where BODS is Installed:

C:\Program Files (x86)\SAP BusinessObjects\InstallData\InstallCache\im.lac.ds.cpdata.cp-4.0-core-64\14.1.1.210\actions\data

Currently "No Files are Present".

Download the latest "SBOP DQM CLEANSING PACK 4.X" as per your Operating System from SAP Market Place.

and Paste to that Location

"C:\Program Files (x86)\SAP BusinessObjects\InstallData\InstallCache\im.lac.ds.cpdata.cp-4.0-core-64\14.1.1.210\actions\data"

Then go to Control Panel and click on Modify of SAP Business Objects Data Services then these Data Cleansing Option is "Activated"

Path:

C:\Program Files (x86)\SAP BusinessObjects\InstallData\InstallCache\im.lac.ds.cpdata.cp-4.0-core-64\14.1.1.210\actions\data

Paste the Downloaded Person Data in that Location:

Go to Control Panel and modify the SAP Business Objects Data Services 4.X.

Currently it was in disable mode because no Cleansing package are available on BODS 4.X Server

Below status after you save the "SBOP DQM CLEANSING PACK 4.X" package "Cleansing package" option are enabled.

After that Cleansing Package Data is Enabled

Start the Installation and Activate Cleansing Package Data

Now Open the SAP Business Objects Data Services Designer 4.X

Open any Jobs to check Data Cleansing Transform activation status

Click on Data Cleanse transform and check the Cleansing Package Name Status

Previously Cleansing package name is "Blank"

Now validate and execute the Job

Now "SBOP DQM CLEANSING PACK 4.X" is activated on Server.

Hi All

Netezza to Teradata Migration

The scope of the document is to detail the design of steps involved to repoint the existing BODS  jobs to Teradata, which are currently pointing to Netezza.

Why there is need to migrate from Netezza to Teradata?

Netezza is one of the key platforms that are serving as a reporting application for BGS. As a part of the BGEDW initiative, Netezza is the first data warehouse that is chosen for the migration to Teradata platform.

The other main reason for the migration of Netezza application to Teradata is the long lasting performance issues in Netezza that is making the reports to be available for Business very late in the day.

BODS job flows (actually used in NZ to TD migration)

In this method, we will be creating BODS jobs for transferring historical data from NZ to TD layer.

Steps of approach for historical loading:

1)    Prepare metadata for source and target tables

The source and target table metadata needs to be defined. The target metadata is in Teradata layer and can be obtained by converting the datatypes and formats as converting Netezza table names to Teradata table names.

This can be achieved by the Netezza-to-Teradata DDL convertor tool which takes the source DDL (NZ) as input and gives the target DDL (TD). Define mappings between the source and target tables at column level

2)    Build BODS jobs for data transfer

Teradata Parallel Transporter with fast load option in replace mode is used to load the Teradata tables with 1:1 mapping in BODS job flow built for this purpose. A single BODS job can be built and then with the help of macros we can replicate the job for any number of tables we want to migrate with 1:1 mapping based on the column position.

3)    Perform migration

Populate tables in target by running the BODS jobs. We need to run the BODS job by passing the required parameters to populate the target tables. This whole process can be automated and run for desired number of times to migrate all the required tables for historical loading.

For Example:

How to migrate from Netezza to Teradata?

Source:  BODS jobs extract data from various source systems and create flat files based upon them.

Arrival: The flat files created as part of the source process will be placed into a landing area referred to as the ‘arrival’ area.

Processing: Various BODS data flows load the data into the staging area with transformations.

Staging: Data is loaded in from the processing area and transformations are undertaken for history handling.

ODS/WH: Various BODS processes apply business rules against the data in the staging databases.

Data Marts: Various BODS processes apply business rules against the data in the staging databases.

To implement the above approach, we will divide the action items in 4 levels:

a)    Source to Archive

b)    Processing to Staging

c)    Staging to Warehouse

d)    Warehouse to Data Mart (Server Specific)

a)   Source To Archive:

1)    Import the job into the repository.

2)    The existing dataflow DF_CONTROL will be replaced with a new work-flow named W_INSERT_DF_CONTROL.

3)    Assign the values to the parameters defined for W_INSERT_DF_CONTROL workflow

4)    The DF_CONTROL table will be re-imported into the ADMIN datastore.

5)    Open the Main dataflow (e.g. D_SRC_ARV_UABSCON) and change the target table properties for the DF_CONTROL by setting the Auto Correct Load = YES under the Advanced Target Table properties.

6)    One ‘source to archive’ work flow (W_SRC_ARV_UABSCON) is shown in the below figure.

b)  Processing to Staging:

1)    Import the job into the repository.

2)    The existing dataflow DF_CONTROL will be replaced with a new work-flow named W_INSERT_DF_CONTROL.

3)    Assign the values to the parameters defined for W_INSERT_DF_CONTROL workflow.

4)    All the tables that will be used have to be re-imported from their respective datastores.

5)    The W_RECOVERY_STG work-flow will be updated by replacing the double dot ‘..’ identifier after the database name with a single dot ‘.’ e.g. [$$SP_DB_EDW_ADMIN_DATABASE]..DF_CONTROL in the workflow W_RECOVERY_STG is changed to [$$SP_DB_EDW_ADMIN_DATABASE].DF_CONTROL.

6)    If required, we will have to add the Post-Load and Pre-Load SQL commands obtained from the migration inventory. The migration inventory SQL commands will have to be converted to Teradata SQL first. Change the Bulk Loader Options for the target table as follows

i.        Bulk Load = Parallel Transporter.

ii.        File = Named Pipe and access module

iii.        Operator = If data already exists - Update else Load.

iv.        Number of Instances = 1.

v.        Mode = Append.

vi.        Bulk Operation = Insert.

vii.        Field Delimiter = 1/27

Named Pipe Parameters

The default value for all the parameters is C:\Program Files\Business Objects\BusinessObjects Data Services\Log\BulkLoader. Change it as below:

viii.        Logdirectory = [$$SP_LOG_DIRECTORY]

ix.         FallbackDirectory = [$$SP_LOG_DIRECTORY]

7)    Change the datatype of the parameter [$P_DATETIME_ACTIVITYDATE] =VARCHAR (19) from DATETIME. Enclose in single quotes, when it is used in Pre/Post-Load commands.

            One processing to staging work flow is shown in the below figure.

             c)   Staging to Warehouse:

1)    Import the job into the repository.

2)    The existing dataflow DF_CONTROL will be replaced with a new work-flow named W_INSERT_DF_CONTROL.

3)    Assign the values to the parameters defined for W_INSERT_DF_CONTROL workflow $P_W_VARCHAR_JOBNAME = $L_VARCHAR_JOBNAME

4)    All the tables that will be used have to be re-imported from their respective datastores.

5)    The W_RECOVERY_STG work-flow will be updated by replacing the double dot ‘..’ identifier after the database name with a single dot ‘.’. e.g. [$$SP_DB_EDW_ADMIN_DATABASE]..DF_CONTROL in the workflow W_RECOVERY_STG is changed to [$$SP_DB_EDW_ADMIN_DATABASE].DF_CONTROL.

6)    Change the datatype of the parameter [$P_DATETIME_ACTIVITYDATE] =VARCHAR (19) from DATETIME.

7)    If required, we will have to add the Post-Load and Pre-Load SQL commands obtained from the migration inventory.

8)    While the load is happening from Teradata to Teradata tables (BODS extracts data, performs transformations, then loads), change the Bulk Loader Options as below:

9)    If any custom SQL is used for the lookups, it has to be modified to make it Teradata compatible. Refer section 10.2 for SQL conversion from Netezza to Teradata.

10)  Open the Target Table Editor and go to Options tab and change the Column Comparison = Compare by position in the General settings.

              One Staging to Warehouse Work Flow is shown in the below figure.

     d)    Warehouse to Server Specific (Data Mart)

1)    Import the job into the repository.

2)    The existing dataflow DF_CONTROL will be replaced with a new work-flow named W_INSERT_DF_CONTROL.

3)    Assign the values to the parameters defined for W_INSERT_DF_CONTROL workflow

4)    All the tables that will be used have to be re-imported from their respective datastores.

5)    The W_RECOVERY_SS work-flow will be updated by replacing the double dot ‘..’ identifier after the database name with a single dot ‘.’. e.g. [$$SP_DB_EDW_ADMIN_DATABASE]..DF_CONTROL in the workflow W_RECOVERY_SS is changed to [$$SP_DB_EDW_ADMIN_DATABASE].DF_CONTROL.

6)    Change the datatype of the parameter [$P_DATETIME_ACTIVITYDATE] = VARCHAR (19) from DATETIME

7)    Add the Post-Load and Pre-Load SQL commands

        One Warehouse to Service Specific work flow (W_WH_SS_PROD_MVMT_HMOVE) is shown in the figure below.

Maestro/ Tivoli Workload Scheduler(TWS)- Batch

Introduction

Tivoli Workload Scheduler (TWS) provides the backbone for automated workload management and monitoring. Offering a single console, real-time alerts, reports and self-healing capabilities, this is software automation to help manage the workloads that span your enterprise.

Scheduler Jobs Process

In order to offer greater flexibility, improved throughput and concurrency, the existing schedules shall be updated to execute BODS processes at the atomic level. In essence, this means that each dataflow in BODS shall have a corresponding job in TWS.

Current Structure:

The current strucuture with BODS Jobs structure hierarchy with Netezza is shown below

So, we can have ‘n’ number of workflows in one job and similarly ‘n’ number of dataflows in one workflow, thereby the relation being One-to-Many for Job:Workflow and One-to-Many for Workflow:Dataflow.

In this case the dependency of one workflow would be on the successful completion of all the dataflows created under it and similarly a whole job would be successful with the completion of all the workflows below that job in the hierarchy.

Proposed structure of BODS Jobs with Teradata

So, we can have only one workflow in one job and ‘n’ number of dataflows in one workflow, thereby the relation being One-to-One for Job:Workflow and One-to-Many for Workflow:Dataflow.

In this case, the dependency of one workflow would be on the successful completion of all the dataflows created under it and similarly a whole job would be successful with the completion of just one workflow created under that job in the hierarchy.

By following the above process, we can migrate from Netezza to Teradata using Business Objects Data Services.

Thanks & Regards,

Vishakha Nigam

The following tutorials have been developed to help you find answers to the most common questions about using SAP BusinessObjects Data Quality Management 4.0, version for SAP. Please refer to the User guide for the detailed steps.

Method 1 – Recommended one: Using the CTS (Transport)

Prerequisite:

• Identify the correct R900xxx.xxx and K900xxx.xxx to use, for it, read the readme.txt file in the following folder <install_dir>/Business Objects Data Services/admin/R3_functions/transport
• In case of DataServices version before 4.1, you need to check the Unicode or Non-Unicode version of SAP system. To do it, log on to the SAP system, in the Menu Bar, go to ->System -> Status:

Step N°	Screenshot	Comment
1		On the SAP Server: Copy R900xxx.xxx in /usr/sap/trans/data, check write privilege on file (chmod 666 on Unix)     Copy K900xxx.xxx in /usr/sap/trans/cofiles, check write privilege on file (chmod 666 on Unix)     With Transaction AL11 you could check that both files are present
2		Launch STMS transaction in SAP Click on , DataServices Transport are not available
3		In Menu Bar, click on    ->Extras       ->Other Requests          ->Add
4		Select the DataServices transport to add. If files copied in folders are: x900yyy.R63, transport request is R63K900yyy. Click on .
5		The DataServices transport is now available, Select it and click on .
6		Fill the right Target Client   Mark checkbox “Ignore Invalid Component Version“   Click on .
7		Open SE37 transaction and check if function module starting with /BODS/* are available. Function module name are displayed in DataServices documentation.

Method 2 - Use .txt file

            (See DataServices documentation)

In some SAP environment, it is frequent to meet a technology called Message Server or SAP Logon Group to load balanced or distribute users and jobs on several SAP application server.

It is possible to create DS datastore that follow the same way.

The Architecture should be similar to:

For connection through Message Server. The SAP datastores in DataServices could only be done through a sapnwrfc.ini file.

The following tasks will be done on the DS server

In the SAP datastore you should now use:

In addition you could setup the RFC destination in the DataServices Management Console.

Here, only application servers could be used, no message server.

It means that you have to add one RFC connection by application server present in the SAP landscape:

This paper combines all the steps from the BI4.x, DS 4.x, FIM 10 and Intercompany 10 Administrator’s Guide with the latest best practices and all the latest SAP KBAs regarding vintela, kerberos and java AD configuration. It is specifically written for Platform 4.x and will not work with earlier versions of XI.

Assumptions & Prerequisites

·         Review the BI4 and DS4 Product Availability Matrix (PAM) to ensure your client and server operating systems, browsers and Active Directory versions are supported.

·         Windows AD authentication only works if the CMS is run on Windows. For single sign-on to the database, the reporting servers must also run on Windows. These Windows machines must be joined to the appropriate AD domain.

·         For multiple AD Forest environments please refer to KBA 1323391

Section 1 – Configure IPS / BI Platform  for AD authentication

Refer to KBA 1631734 to configure Active Directory for IPS / BI Platform and Tomcat server. Following are the high level steps covered by this KBA

i)         AD Service Account Configuration on Windows AD server

ii)        Configure the AS plugin on CMC

iii)       Configure all services to run on new service account

iv)       Configure manual AD to Java Application Servers (Tomcat)

v)        Configure BI Launchpad and CMC for manual AD

vi)       Configure Active Directory SSO on Tomcat

The following sections focus on the additional configurations required to enable AD authentication and SSO (where supported) on client components for BI and EIM 4 stack.

AdditionalTips: -

a)   SSO will not work from the Tomcat Server, use client machine for tests

b)   Create SPNs for all application URLs for hostnames and FQDN (Fully Qualified Domain Names)

c)   If load balancer is configured, create SPN for all URLs of load balancers (including FQDNs)

d)   If there are multiple domain controllers for AD authentication, configure all of them in the [Realms] section of krb5.ini

e)   If you are not able to locate Tomcat properties shortcut, run the following command on command line to open the configuration (Example is for Tomcat6)

Tomcat6w //ES/<Service Name>

Where <Service name> can be found from windows services i.e. the name under which Tomcat service is running. By default the name is BOEXI40Tomcat

f)    In multi-server architecture, KBA needs to be applied on all the machines running IPS / BI platform and application servers

g)   Not all products support SSO, however most of them support Windows AD. Refer to product admin guides for details

h)   SSO does not work for CMC for security reasons, however manual AD is supported

Section 2 – Configuring BI Clients for AD Authentication

In Section 1, we have already configured the CMC and BI Launchpad to run for AD and SSO. This section focuses on remaining client tools (where additional configuration is required)

Information Design Tool (IDT) AD Configuration

On each client machine, navigate to InformationDesignTool.ini at the following path

<LINK_DIR>\SAP BusinessObjects Enterprise XI 4.0\win32_x86

Add the following configuration (Ensure that IDT application is not running)

-Djava.security.auth.login.config= c:\windows\bscLogin.conf

-Djava.security.krb5.conf=c:\windows\krb5.ini

Note: - While running the application, enter the SYSTEM name and Authentication method to ‘Windows AD’, leaving ‘User Name’ and ‘Password’ blank

Universe Designer AD Configuration

No additional configuration is required. While running the application, enter the SYSTEM name and Authentication method to ‘Windows AD’, leaving ‘User Name’ and ‘Password’ blank.

Web Intelligence Rich Client AD Configuration

No additional configuration is required. While running the application, enter the SYSTEM name and Authentication method to ‘Windows AD’, leaving ‘User Name’ and ‘Password’ blank.

LCM (Life Cycle Management) SSO Configuration

Create a file LCM.properties at the following path

<LINK_DIR>\Tomcat6\webapps\BOE\WEB-INF\config\custom\

Update the file with the following contents

1. authentication.visible=true
2. authentication.default=secWinAD
3. cms.default=<cmc-name>:<cmc-port>

Note: - Tomcat restart is required for the changes to be effective and perform the changes on all application servers for .properties file change

Section 3 – Configuring Data Services 4.x and FIM 10 for AD Authentication

Data Services Designer AD Configuration

No additional configuration is required. While running the application, enter the SYSTEM name and Authentication method to ‘Windows AD’, leaving ‘User Name’ and ‘Password’ blank.

Data Services Management Console AD Configuration

Ensure that Section 1 is performed on all the application server machines where DS Management Console application is running

FIM AD Configuration

Navigate to FIM (Financial Information Management) administration console web application and select the option ‘Configure Financial Information Management Server’.

Check the option ‘Show Authentication Mode’. This will enable FIM application to show additional drop down for selecting the authentication mode