Bulk import performance

Hi Folks,
We're getting ready to migrate our first teams into our new Vibe environment but I'm wondering if we can do anything to improve the performance of bulk imported documents (drag and drop into the webclient).
Our environment: Vibe 3.3 on SLES11 SP2 (3 virtual servers)
Server 1 (Tomcat):
2x CPU
8GB RAM (Java cache 6GB)
Server 2 (Lucene):
2x CPU
8GB RAM (Java cache 6GB)
Server 3 (MySQL):
2x CPU
4GB RAM
When I do a bulk import, files import and everything is fine but if I'm concerned about the time it takes. During my test imports the only process that has any significant utilization is Java on the Tomcat server, but it bounces from between 30-90% on a single CPU. If needed I can easily crank these VMs up to 8 CPUs and 64GB RAM if needed. That said, if I can't tax the current servers, there's no need to increase resources.
Does anyone know a way to improve the import performance? I want to be able to peg all my CPUs at 100% if need be.

Hi John,
many thanks for your reply. Import and gathering stats is done through an application. I have only the following information about what is supposed to happen when we run the statistic gathering process:
"Optimization - Uses sub-commands for database optimization (DBMS_STATS*). These are important for the Oracle optimizer to generate optimal execution plans.
Optimize Schema - Runs the DBMS_STATS.GATHER_SCHEMA_STATS(USER,cacade=>true) command. This procedure gathers (not estimates) statistics for all objects in a schema and on the indexes.
Optimize the schema after you import a industry model from a dump file, and run the Optimize command whenever you have extensive changes to the data, such as imports or major updates.
Optimize Feature Classes - Runs the DBMS_STATS.GATHER_TABLE_STATS([USER], [Table], cascade=>true) command for the feature class tables. This procedure gathers table, column, and index statistics."
The application we use allows to the gather statistics in two different places. I now realise that we have only used one of the two so far and if my understanding of the documentation is right the one we have used does not gather all statistics. With your explanation the observed behavouir makes sense. Next time I will gather statistics using the second functionality to see if that one gathers all statistics at once.
Many thanks again, Rob

Similar Messages

How to use QPXVBLK: QP: Bulk Import of Price List?

hi,
i'm trying to create a script for import price lists. i've read that this concurrent program "QPXVBLK: QP: Bulk Import of Price List" is what i need. i've been able to successfully create the script to populate the interface tables:
QP_INTERFACE_LIST_HEADERS
QP_INTERFACE_LIST_LINES
QO_INTERFACE_PRICING_ATTRIBS
but i'm getting an error.
3436374/PRL/QP_INTERFACE_LIST_HEADERS/24934 Cannot perform Insert, as record with orig_sys_header_ref 24934 already exists for the list_source_code .
i figured it has something to do with the ORIG_SYS_HEADER_REF, ORIG_SYS_LINE_REF and ORIG_SYS_PRICING_ATTR_REF columns.
below are values i'm currently passing.
QP_INTERFACE_LIST_HEADERS TABLE
- ORIG_SYSTEM_HEADER_REF = LIST_HEADER_ID
QP_INTERFACE_LIST_LINES TABLE
- ORIG_SYS_LINE_REF = LIST_LINE_ID
- ORIG_SYS_HEADER_REF = LIST_HEADER_ID of parent record in QP_INTERFACE_LIST_HEADERS TABLE
QP_INTERFACE_PRICING_ATTRIBS TABLE
- ORIG_PRICING_ATTR_REF = PRICING_ATTRIBUTE_ID
- ORIG_SYS_LINE_REF = LIST_LINE_ID of parent record in QP_INTERFACE_LIST_LINES TABLE
- ORIG_SYS_HEADER_REF = LIST_HEADER_ID of parent record in QP_INTERFACE_LIST_HEADERS TABLE
if these are not correct, what values should be passed on this columns?
thanks
allen

hi all,
ii would like to add a follow up question on this.
what fields are required when inserting records into the interface tables for the QP: Bulk Import of Price List?
i've been running some tests and while some price lists are imported with no issues, some are not and some only partial. i'm not receiving any errors on the import which makes this harder for me to debug.
any thoughts?
thanks.
allen

Is there a way to bulk import/edit SMTP Proxy Adresses in Unity Connection 10?

I'm looking for a way to perform a bulk import of SMTP Proxy Addresses. BAT doesn't seem to have a way to do it. I have 3,000 users, not looking forward to doing it manually! Any suggestions?
Thanks

Is there a reason you think it wont work with BAT? BAT has a field called 'SmtpProxyAddresses' which can be used to populate it. You will have to export your existing user data, massage it a little and add the SMTP Proxy address and use the Update option to make the changes. You can also set the "Generate SMTP address......" in the template for new users.
http://www.cisco.com/c/en/us/td/docs/voice_ip_comm/connection/8x/user_mac/guide/8xcucmacx/8xcucmacappa.html

Account Relationships. Is it possible to do bulk imports?

Hi,
I would like to know if it´s possible to import csv files with information about Account Relationships. I need to create different types of accounts and then I need to import relationship between them (For instance: Company A will be a Suplier of CompanyB or CompanyB will be a customer of CompanyC).
Is there any way to do bulk imports with this type of information?
Also, what about reporting, is it possible to do reports or extractions of data to obtain this info?
Thanks in advance

Hi Antonio_A,
Just in case you are looking for an alternative to CSV upload, you can actually perform the imports of Account Relationships via Web Services.

I'm currently using Adobe Acrobat X to create PDF portfolios for client deliverables. Within the portfolio, there is a "Files" view which can be seen below. I'm interested in knowing if it's possible to bulk import metadata into either the Description fie

I'm currently using Adobe Acrobat X to create PDF portfolios for client deliverables. Within the portfolio, there is a "Files" view which can be seen below. I'm interested in knowing if it's possible to bulk import metadata into either the Description field or any user created field via a CSV, spreadsheet, etc. Thanks for any insight which can be given.

I'm currently using Adobe Acrobat X to create PDF portfolios for client deliverables. Within the portfolio, there is a "Files" view which can be seen below. I'm interested in knowing if it's possible to bulk import metadata into either the Description field or any user created field via a CSV, spreadsheet, etc. Thanks for any insight which can be given.

Error in QP: Bulk Import of Price List

Hi All,
I am trying to import price list records using the concurrent program 'QP: Bulk Import of Price List'. I am facing one issue in this...below is the error detail:
7164262/PRL/QP_INTERFACE_LIST_LINES//RAGH-L-3 A price break child line must have at least one pricing attribute.
The header and Qualifier records are getting imported sucessfully, the issue is with the line and it's attributes. Below are the fields on interface tables whcih i m populating:
INSERT INTO QP_INTERFACE_LIST_LINES (
ORIG_SYS_LINE_REF,
ORIG_SYS_HEADER_REF,
LIST_LINE_TYPE_CODE,
START_DATE_ACTIVE,
END_DATE_ACTIVE,
ARITHMETIC_OPERATOR,
OPERAND,
PRIMARY_UOM_FLAG,
PRODUCT_PRECEDENCE,
INTERFACE_ACTION_CODE,
PROCESS_FLAG,
PROCESS_STATUS_FLAG,
PRICE_BY_FORMULA,
AUTOMATIC_FLAG,
REPRICE_FLAG
--PRICING_GROUP_SEQUENCE
) VALUES
'RAGH-L-3',
'RAGH-H-3',
'PLL',
sysdate,
NULL,
'UNIT_PRICE',
150,
'Y',
230,
'INSERT',
'Y',
'P',
NULL,
'Y',
'Y'
--0
INSERT INTO qp_interface_pricing_attribs
orig_sys_pricing_attr_ref,
orig_sys_line_ref,
orig_sys_header_ref,
PRODUCT_ATTRIBUTE_CONTEXT,
product_attr_code,
product_attr_val_disp,
product_uom_code,
interface_action_code,
process_flag,
process_status_flag,
PRICING_ATTRIBUTE_CONTEXT,
PRICING_ATTRIBUTE,
PRICING_ATTR_VALUE_FROM_DISP,
COMPARISON_OPERATOR_CODE,
PRICING_ATTR_VALUE_TO_DISP )
VALUES
'RAGH-ATTR-3',
'RAGH-L-3',
'RAGH-H-3',
'ITEM',
'INVENTORY_ITEM_ID',
'CM13139',
'EA',
'INSERT',
'Y',
'P',
'ADS_PRICING',
'COST',
'25',
'BETWEEN',
'30' );
Using the above process I am getting the error as mentioned:"7164262/PRL/QP_INTERFACE_LIST_LINES//RAGH-L-3 A price break child line must have at least one pricing attribute.".
If I am not using the pricing attributes PRICING_ATTRIBUTE_CONTEXT,PRICING_ATTR_CODE, Pricing_attr_value_from,COMPARISON_OPERATOR_CODE, PRICING_ATTR_VALUE_TO I am able to import the line records also.
The script provided by Oracle also uses the same columns in the interface table, kindly help me if any one have any idea regarding this issue.

Hi Luko,
I tried the same script given by oracle, but it is giving me the error as I mentioned earlier. When I am not using the pricing line attributes, it is working fine. The error is coming only when I am passing the pricing attribute parameter(PRICING_ATTRIBUTE_CONTEXT, PRICING_ATTR_CODE, PRICING_ATTR_VALUE_FROM_DISP,
COMPARISON_OPERATOR_CODE, PRICING_ATTR_VALUE_TO_DISP).
I am passing the values as shown below:
PRICING_ATTRIBUTE_CONTEXT-Code fro pricing context i.e PRICING ATTRIBUTE
PRICING_ATTR_CODE-code for attribute i.e. FREIGHT_COST
PRICING_ATTR_VALUE_FROM_DISP-200
PRICING_ATTR_VALUE_TO_DISP-300
COMPARISON_OPERATOR_CODE-BETWEEN
Thanks!

Bulk import error - Missing required attribute fullname

Folks,
I am having trouble trying to bulk import some users.
I have the following file for import (cut down from what I really want to import):
command,user,waveset.resources,password.password,password.confirmpassword,global.firstname,global.lastname,global.fullname
Create,user1,myLDAPresourcename,passwd,passwd,myfirstname1,mysurname1, myfirstname1 mysurname1This works using Accounts->Launch Bulk Actions with Action set to From Action List and everything else on that page left at the default value. The correct data appears in IDM and the LDAP resource.
Now if I try to import the following file, in which the user does not have a first name, I get in to trouble:
command,user,waveset.resources,password.password,password.confirmpassword,global.firstname,global.lastname,global.fullname
Create,user2,myldapresourcename,passwd,passwd,,mysurname2,mysurname2
The error message is:
com.waveset.Util.WaveSetException: An error occurred adding user uid=user2,.. to resource myLDAPresourcename.
com.waveset.util.WavesetException: Missing required attribute fullnameClearly the import file contains a column for fullname (column 8) and a value for fullname in the only row of data mysurname2. As far as I am concerned it is perfectly valid for a user not to have a firstname; but they must have a fullname.
I have tried using an import file without a value for fullname, but with a firstname and lastname like this
command,user,waveset.resources,password.password,password.confirmpassword,global.firstname,global.lastname,global.fullname
Create,user3,myLDAPresourcename,passwd,passwd,myfirstname3,mysurname3,This works. The correct data appears in IDM and the LDAP resource and IDM has generated a fullname value of {Firstname} {Surname}. Trying to leave out the firstname and the fullname gives the same error as above.
The configuration of my resource (in summary) is
objectclasses:
     top
     person
     inetOrgPerson
Account Attributes:
     accounted     -> uid               [required]
     password     -> userPassword
     firstname     -> givenname
     lastname     -> sn               [required]
     objectClass     -> objectClass
     fullname     -> cn               [required]
As an experiment I have tried reconfiguring the resource so that fullname is not marked as required.
Importing user1 and user3 still works and still creates the correct data in IDM and the LDAP resource.
Importing user2 now gives a different error:
com.waveset.Util.WaveSetException: An error occurred adding user uid=user2,.. to resource myLDAPresourcename.
javax.naming.directory.SchemaViolationException: [LDAP: error code 65 Object Class Violation]I dont understand why, during the import of a user without a firstname IDM will not generate a fullname or use the supplied fullname.
Can anyone help?
Regards,
Gavin

After further testing it turned out that I couldn't create a user without a firstname from the IDM GUI and this in turn was because of the way fullname was generated by the user form.
I've got the user form changed and now user creation (from the GUI and via bulk import) is working as hoped.

How to bulk import data into CQ5 from MySQL and file system

Is there an easy way to bulk import data into CQ5 from MySQL and file system? Some of the files are ~50MB each (instrument files). There are a total of ~1,500 records spread over about 5 tables.
Thanks

What problem are you having writing it to a file?
You can't use FORALL to write the data out to a file, you can only loop through the entries in the collection 1 by 1 and write them out to the file like that.
FORALL can only be used for SQL statements.

Trouble creating a csv file to bulk import users

Hi Guys,
I'm trying to put a .csv file together to bulk import users. I'm practicing with one user right now. This is my CSV header:
GivenName,Name,SamAccountNAme,Description,Department,OU,Enabled,Accountpassword,PasswordNeverExpires,CannotChangePassword
This is my first line:
newuser,newuser,newuser,newuser,Group2,OU=Group2,DC=CONTOSO,DC=COM,$True,P@ssw0rd,$True,$True
When I test my file this is the output:
GivenName                      : newuser
Name                               : newuser
SamAccountNAme            : newuser
Description                       : newuser
Department                      : Group2
OU                                    : OU=Group2
Enabled                            : DC=CONTOSO
Accountpassword             : DC=COM
PasswordNeverExpires    : $True
CannotChangePassword : P@ssw0rd
I obviously need to state the OU in the csv otherwise because the values are not in the correct place right now. Putting OU=Group2,DC=CONTOSO,DC=COM between brackets "..." doesn't help. Then all values are stated at 'GivenName'.
Please help:-)

Thank you Mahdi,
When I do that, I get this:
GivenName            : newuser,newuser,newuser,newuser,Group2,"OU=Group2,DC=contoso,DC=com"$True,P@ssw0rd,$True,$True
Name                 :
SamAccountNAme       :
Description          :
Department           :
Path                 :
Enabled              :
Accountpassword      :
PasswordNeverExpires :
CannotChangePassword :
There is one (,) missing between path and $true. I tested it and it works fine. However I am glad you found your way.
Mahdi Tehrani |
|
www.mahditehrani.ir
Please click on Propose As Answer or
to mark this post as
and helpful for other people.
This posting is provided AS-IS with no warranties, and confers no rights.
How to query members of 'Local Administrators' group in all computers?

Maintain Sequence During Bulk Import

I'm doing 'Mass Import' in transaction SOLMAN_WORKCENTER->START_IMPORT for a CTS project. But, I noticed that it triggers all the transports at once and import depends on the size of objects in the transport and not sequence.
For example, if I've the following TRs in sequence and each TR is dependent on the previous one.....
1 TR1
2 TR2
3 TR3
If I trigger Bulk import for these TRs and TR1 has bulky objects, TR2 and TR3 will be imported before the dependent TR1, which will cause failure in import.
Is there any way to trigger a mass import sychronously? So that TR2 will be moved only if TR1 is successful and TR3 will be moved only after if TR2 is successful.
I'm facing the same issue in imports. The only solution to this problem was to move the TRs manually in STMS, one by one as per the sequence. It worked. But, it's not feasible for scenarios where the no of TRs in queue for imports are quite high.I checked the SAP help as well, but got no information on the same.
Kindly suggest.

Hello.
Would you answer the following question:
Have you already developed the program for transport requests import into a target (especially Productive) system ?
With the possibility to maintain the requests sequence ?
If so, then inform me, pls., how.
Best regards,
Vlad Beloglazov

VG310 Bulk Import issues

I'm trying to bulk import a VG310 SCCP into CUCM 10.5.2 and am having an issue. I've tried everything I can think of but nothing works. I know I'm close because it takes the first entry in the csv and puts it in but then errors on any of the rest with the following error:
SKIGWC288264458(Record Number 2) 2036 DOMAIN NAME MAC Address already exists.
SKIGWC288264458(Record Number 3) 2036 DOMAIN NAME MAC Address already exists.
Based on that I am assuming it doesn't like something about the MAC address being the same in the Domain Name field in the csv. I've tried leaving the rest of the entries blank but it doesn't like that. I tried adding 000 to the end of it (the way it adds the AN name to the port) and then incrementing up from there (001,002,etc) but that just created 24 new gateways.
Any help is greatly appreciated. I've spent a lot of time messing with this. I would just put them in manually but I've got several VG350's to program as well and fear I will run into the same issue. I'd love to get it cleared up so as to automate the process somewhat.
I'm attaching a copy of the BAT I created for reference.
Thanks!

Ayodeji, thank you for your input.
I took the file you attached and modified it with my info specific to the site I'm programming. I had tried doing it like this previously but got an error:
SKIGWC288264458(Record Number 1) 2004 GATEWAY NAME is blank.
SKIGWC288264458(Record Number 2) 2004 GATEWAY NAME is blank.
It gave me this same error when I used yours. I have the base gateway itself configured in CUCM already with all blank endpoints and uploaded using the file you sent as an example. As you can see, it appears not to like something with the field GATEWAY NAME. I've tried changing this to DOMAIN NAME in mine and that is the one that I found where it would work for the first port only, but only after paring down several other fields. Directory Number 1 in yours has to be PORT DIRECTORY, LINE DESCRIPTION 1 has to be just LINE DESCRIPTION. The same holds true for all the rest of the fields that have a 1 in them from what I was able to tell. In the example I'd previously attached, the first port was programmed when I imported but the rest errored with DOMAIN NAME MAC Address already exists error I noted in the original post.
I am attaching the adjusted file I tried using based on yours. Please let me know what you think.
Thanks again for the reply.

Bulk Import Projects along with Project Plan

Hi,
I am working on a requirement to bulk import Projects into Ms Project Server along with associated Project Plans. I have checked Solution Starter Bulk Create source code. However, the source code doesn't check-in project plans associated with
projects.
It just creates Projects. I am looking for approach and alternatives to implement this requirement. Any help in this regard will be appreciated.

Hi,
From experience you will be able to create Bulk project up to Project details only , for schedule level consider using PSI to Add tasks under the projects.
Regards, Syed Faizan ur Rehman, CBPM®,PRINCE2®, MCTS

OWA - Bulk Import of Contacts into Group

What is an option for importing a list of email addresses into OWA Contact Group/creating OWA Contact Group and pasting members' mail addresses into it? Similarly, if I create a new folder in My Contacts, how I can bulk import into this new folder? When
I use Import Contacts standard OWA feature, contacts always imported into default Contacts folder?

Hi,
Based on my knowledge, there is no feature available in OWA 2013 for you to import Contacts. Please refer to the following article to import Contacts or Contact Group from Outlook 2013 to have a try:
http://office.microsoft.com/en-001/outlook-help/import-contacts-HA102919678.aspx
Regards,
Winnie Liang
TechNet Community Support

Bulk import of .ics attachments

Hi all,
My IMAP account lives on a Novell Groupwise system, and I maintain a calendar and to-do list in that system. When viewed from Mail.app though, the Calendar and Checklist folders contain mail messages with .ics attachments in them. Mail correctly parses these as iCal events, and turns them into iCal meetings or to-do items.
Is there a way of telling Mail that certain folders contain this kind of information, and to import them automatically? Or is there some means of bulk-importing the .ics attachments, or scheduling this to happen?
Cheers,
James

I used the mac groupwise client to set up rules to sort the mail versus other types of messages. I don't use Ical for may calendar but once you have them sorted maybe you can do something with them in apples mail program.
Can you schedule appointments with other groupwise users with Ical. I did some reading and discovered that groupwise did not support the webcal standard so thought it would not work?

QP: Bulk Import of Price List

Hi All,
I encouter this error when i run this CP.
Program Name : ‘QP: Bulk Import of Price List’
Erroring with the following error :
QP_INTERFACE_LIST_HEADERS/SAMPLE_HEADER Invalid Date.
Output :
------Processing Price List Entity--------
-----------Bulk Pricelist Data Loader---------
Paremeters
Entity Name:
Process Id:
Process Type:
Process Parent: Y
Number Of Threads: 1
Number Of succesfully Processed Headers: 0
Number Of Errored Headers: 1
Number Of succesfully Processed Qualifiers: 0
Number Of Errored Qualifiers: 0
Error Details
4766507/PRL/QP_INTERFACE_LIST_HEADERS/SAMPLE_HEADER Invalid Date.
4766507/PRL/QP_INTERFACE_LIST_HEADERS/SAMPLE_HEADER Invalid Date.
thanks,

Please check the date format mentioned for start_date_active
For QP_INTERFACE_LIST_HEADERS, it is VARCHAR2 and should be inserted as a character string in the format YYYY/MM/DD
For QP_INTERFACE_LIST_LINES, it is however a DATE field which means that you can enter a normal date there
Regards
Soumya

Bulk import performance

Similar Messages

Maybe you are looking for