Data Integrator 11.7.3: Error 70602

Question

Hi,
I have scheduled my entire production environment in one job, many levels, many workflows, dataflows, scripts, variables aso. After an upgrade from 11.7.2 to 11.7.3 the job stops every night initiating one particular workflow and dataflow. It's a Sybase repo.
Here is the part of the log where the df initiates and fails. Anybody with the same experience or knows what this sybase error 70602 often relates to in BODI ?
23687     1     WORKFLOW     04.12.2008 06:29:17     Work flow <EDW_FAKTA_OPPDRAG_FEILMELDING_WF> is started.
4139     1     DATAFLOW     04.12.2008 06:29:23     Process to execute data flow <EDW_FAKT_OPPDRAG_FEILMELDING_AKTIV_DF> is started.
4139     1     DATAFLOW     04.12.2008 06:29:29     Data flow <EDW_FAKT_OPPDRAG_FEILMELDING_AKTIV_DF> is started.
4139     1     DATAFLOW     04.12.2008 06:29:29     Cache statistics for data flow <EDW_FAKT_OPPDRAG_FEILMELDING_AKTIV_DF> are not available to be used for optimization and need
4139     1     DATAFLOW     04.12.2008 06:29:29     to be collected before they can be used.
4139     1     DATAFLOW     04.12.2008 06:29:29     Data flow <EDW_FAKT_OPPDRAG_FEILMELDING_AKTIV_DF> using PAGEABLE Cache with <3443 MB> buffer pool.
4139     1     DATAFLOW     04.12.2008 06:31:47     Data flow <EDW_FAKT_OPPDRAG_FEILMELDING_AKTIV_DF> is terminated due to error <70602>.
4139     1     DATAFLOW     04.12.2008 06:31:47     Process to execute data flow <EDW_FAKT_OPPDRAG_FEILMELDING_AKTIV_DF> is completed.
23687     1     WORKFLOW     04.12.2008 06:31:48     Work flow <EDW_FAKTA_OPPDRAG_FEILMELDING_WF> is terminated due to an error <70602>.
23687     1     WORKFLOW     04.12.2008 06:31:48     Work flow <FAKTA_2_KJORING> is terminated due to an error <70602>.
23687     1     WORKFLOW     04.12.2008 06:31:48     Work flow <EDW_WF> is terminated due to an error <70602>.
23687     1     JOB     04.12.2008 06:31:48     Job <DW_Job> is terminated due to error <70602>.
regards,
-jem

Answer 1

Hi Manoj,
Takes some time to test this out because running df's separately does not give the same results or an authentic setting and the test environment is poorly. We are therefore adjusting-watching the prod job, which run every night on the sybase repo. What we did:
- The first night we disabled both quality stats and sample data for all validation objects in the job and the job completed successfully.
- The second night we activated the quality stats for all validation objects in the job and the job completed successfully.
- Then we took out a df to a new and separat job for a stresstest against the repo, running 50 days with data. With only the quality stats still checked. It failed with this error:
13328 1 RUN-050406 13.12.2008 12:42:45 |Session New_Job2|Workflow EDW_FAKTA_ORDRE_KONTERING_WF|Dataflow EDW_FAKT_ORDRE_KONTERING_DF
13328 1 RUN-050406 13.12.2008 12:42:45 Data flow <EDW_FAKT_ORDRE_KONTERING_DF> received a bad system message. Message text from the child process is <blank message,
13328 1 RUN-050406 13.12.2008 12:42:45 possibly due to insufficient memory>. The process executing data flow <EDW_FAKT_ORDRE_KONTERING_DF> has died abnormally. For
13328 1 RUN-050406 13.12.2008 12:42:45 NT, please check errorlog.txt. For HPUX, please check stack_trace.txt. Please notify Customer Support.
13328 1 RUN-050409 13.12.2008 12:42:46 |Session New_Job2|Workflow EDW_FAKTA_ORDRE_KONTERING_WF
13328 1 RUN-050409 13.12.2008 12:42:46 The job process could not communicate with the data flow <EDW_FAKT_ORDRE_KONTERING_DF> process. For details, see previously
13328 1 RUN-050409 13.12.2008 12:42:46 logged error <50406>.
- We then unchecked the quality stats and the job completed successfully.
- Finally we hold the prod job for 4 days, to give more data/stress on the repo (still with only the quality stats checked) and the job completed successfully.
Tonight we will try to turn on sample data for the stage workflow.
But when the job did fail in the beginning it failed on df's in the edw workflow. Means it run through the stage workflow with both quality stats and sample data. (As mentioned earlier post, stage and edw are different schemas in the data warehouse database, oracle 10G. And the stagejobs read from post and the edw from the stage.)
One difference I can see is that the validations in stage are pretty simple and with "clean" ports, read:nice formats. And we are just checking on incoming missing values from source-post (NULL). In the edw df's the validations are more complex and we often test on lookup returns (if-exists variants). And the format on the return values are not always nice..for instance, Number values in oracle (large integer contents) are converted to decimal (28,7).. We have not changed that. But again, if we run these df's manually - not scheduled, it runs through.        
Forgot to mention that the job got 135 df's, and roughly 50 with validations.
I just wonder how robust this feature is..and how does it really works ? Fx how does the good rows impact the traffic against the repo ?..not at all or may huge amount of good data slow down stats counts/updates ?  What about "bad formats" ?..is converting portformats an issue, against the repotables I mean ?
Any parameters to adjust in general ?
I guess sybase is not the most robust platform either.. What about DS 3.0 ? Is it more robust ?
I did also adjust the lenght and type of characters in the validation name. So they are pretty simple now. It's not easy to get any detail traces, in prod and with huge amount of data..but I will try..
Regards,
- jem