Handling Large list with 150+ columns(40+ Searchable) and 100000 rows.

Hi All,
We are working on project architecture for project with the following issue:
1. List having 150+ columns out of which:
- 40+ columns need to have search (i.e Indexing)
- 30+ lookup columns
- 10+ date time columns
- 40+ multiline text columns
2. List having 100000+ records
- As per business we don’t have any specific folder criteria to divide list items into folders.
Business Requirements:
- filter on the list (it should show real time data so can’t use SharePoint Search)
- Export to Excel 30000+ data in <30 Sec
Our list is already normalized and we don’t see any scope to normalize it further.
Problems of breaking the list into multiple list:
1.)Low performance as join is taking very much time(due to 30,000 items)
2.)Secure transaction problem i.e. suppose we broke list into 5 child lists. Now if I need to add an item in list then it has to be implemented by adding part of item to each child list.
There may arise problem such as items inserted to 4 list successfully but failed in insertion into last list.
Thank you in advance.
Thanks, Satish

Hi All,
We solved this problem by normalizing list into many list.
Then creating one search list with all columns as plain text no look up this list have all the columns from all different list. Then writing event receiver to update this search list on item change.
and one timer job which will delete and recreate this search list weakly.
Let me know if you have any better solution.
-Satish

Similar Messages

Handling large messages with MQ JMS sender adapter

Hi.
Im having trouble handling large messages with a MQ JMS sender adapter.
The messages are around 35-40MB.
Are there any settings I can ajust to make the communication channel work?
Error message is:
A channel error occurred. The detailed error (if any) : JMS error:MQJMS2002: failed to get message from MQ queue, Linked error:MQJE001: Completion Code 2, Reason 2010, Error Code:MQJMS2002
The communication channel works fine with small messages!
Im on SAP PI 7.11, MQ Driver is version 6.
Best Regards...
Peter

The problem solved itself, when the MQ server crashed and restarted.
I did find a note that might could have been useful:
Note 1258335 - Tuning the JMS service for large messages or many consumers
A relevant post as well: http://forums.sdn.sap.com/thread.jspa?threadID=1550399

Creating Select List with multiple columns

I want to create a select list with multiple columns. I followed the demo application model described in the by Denes Kubicek (Currently my reference for APEX !!)
The code is as follows:
CREATE OR REPLACE FUNCTION getcrops(p_cropid IN NUMBER)
RETURN VARCHAR2
IS
v_cropid VARCHAR2 (400);
v_fcode VARCHAR2 (400);
v_product VARCHAR2 (400);
v_var VARCHAR2 (400);
v_fname VARCHAR2 (400);
v_acres VARCHAR2 (400);
v_style_start VARCHAR2 (400);
v_style_end VARCHAR2 (400);
v_return VARCHAR2 (400);
BEGIN
FOR c IN (select "CROP"."CROPID" as "CROP ID",
     "CROP"."FIELDCODE" as "FIELD CODE",
     "CARROTPRODUCTLIST"."CARROTPRODUCTNAME" as "PRODUCT",
     "VARIETYLIST"."VARIETYNAME" as "VARIETY",
     "FIELD"."FIELDNAME" as "FIELD NAME",
     "CROP"."SIZEINACRES" as "ACRES"
from     "FIELD" "FIELD",
     "CARROTPRODUCTLIST" "CARROTPRODUCTLIST",
     "VARIETYLIST" "VARIETYLIST",
     "CROP" "CROP"
where "CROP"."CARROTPRODUCTTYPE"="CARROTPRODUCTLIST"."CARROTPRODUCTID"
and     "CROP"."VARIETYID"="VARIETYLIST"."VARIETYLISTID"
and     "CROP"."FIELDID"="FIELD"."FIELDID")
LOOP
v_cropid := TO_CHAR (c.'CROP ID', 'FML999G999G999G999G990');
v_fcode := c.'FIELD CODE';
v_product := c.'PRODUCT';
v_var := c.'VARIETY';
v_fname :=c.'FIELD NAME';
v_acres :=c.'ACRES';
FOR i IN 1 .. 12 - LENGTH (c."CROP ID")
LOOP
v_cropid := v_cropid || ' ';
END LOOP;
FOR i IN 1 .. 12 - LENGTH (c.'FIELD CODE')
LOOP
v_fcode := v_fcode || ' ';
END LOOP;
FOR i IN 1 .. 12 - LENGTH (c.'PRODUCT')
LOOP
v_product := v_product || ' ';
END LOOP;
FOR i IN 1 .. 12 - LENGTH (c.'VARIETY')
LOOP
v_var := v_var || ' ';
END LOOP;
FOR i IN 1 .. 12 - LENGTH (c.'FIELD NAME')
LOOP
v_fname := v_fname || ' ';
END LOOP;
FOR i IN 1 .. 12 - LENGTH (c.'ACRES')
LOOP
v_acres := v_acres || ' ';
END LOOP;
v_return := v_cropid || v_fcode || v_product || v_var || v_fname || v_acres;
END LOOP;
RETURN v_return;
END;
I created this anonymous Pl/SQL function at a application level ..Then I followed up to create a select list with a function inside. However I could not create select list with the command suggested in the demo which is
select getcrops(cropid) d, cropid r from crop;
APEX (2.1) returns a error message. What am I missing ? Should the function be called from somewhere else after creating a regular select list..? Where the functions (Pl/SQL) should be ideally stored for application level access..? and for across application level access ? How can I check the function works good. Help is appreciated.

Still really one column.
If they need to be independent then you've got to have three selection lists.
%

Selection list with multiple columns

How can i create a selection list with multiple columns?

Still really one column.
If they need to be independent then you've got to have three selection lists.
%

Select list with multiple columns

I want to create a select list with multiple column option. What are the options. Any developments in apex 3.0.1 ? Thanks in advance. I tried the tutorials by Kubicek.
I am kind of stuck to see the function entry inside LOV definition returns a error.

Still really one column.
If they need to be independent then you've got to have three selection lists.
%

List with multiple column

hello all ,
how to create a list with multiple columns ? can anyone share some examples or code snippets

What I would usually do is create a movieclip/sprite that contains a row with every column in it (each column being a textfield). Then pile up the rows one atop the other (using Actionscript) as they are populated with data.

How to export All computer list with operating system from AD and all attributes like disable or enable and OU location also?

how to export All computer list with operating system from AD and all attributes like disable or enable and OU location also?
I have tried with dsquery below but status is not showing there.
dsquery * -filter "(objectCategory=computer)" -attr name operatingSystem

last logon user name - not really stored (or lined) with computer object. However, you can get this info during the logon process or from the computer. Here is method:
http://portal.sivarajan.com/2010/07/user-profile-and-os-info-powershell.html
Santhosh Sivarajan | Houston, TX | www.sivarajan.com
ITIL,MCITP,MCTS,MCSE (W2K3/W2K/NT4),MCSA(W2K3/W2K/MSG),Network+,CCNA
Windows Server 2012 Book - Migrating from 2008 to Windows Server 2012
Blogs: Blogs
Twitter: Twitter
LinkedIn: LinkedIn
Facebook: Facebook
Microsoft Virtual Academy:
Microsoft Virtual Academy
This posting is provided AS IS with no warranties, and confers no rights.

Handling large files with FTP in XI

Hi All,
I have a file scenario where i have to post the file with size of more than 500MB and with the fields upto 700 in each line.
The other scenario worked fine if the file size is below 70MB and less number of fields.
Could anyone help me in handling such scenario with large file size and without splitting the file.
1) From your previous experience did you use any Tools to help with the development of the FTP interfaces?
2) The client looked at ItemField, but is not willing to use it, due to the licensing costs. Did you use any self-made pre-processors?
3) Please let me know the good and bad experiences you had when using the XI File Adapter?
Thanks & Regards,
Raghuram Vedagiri.

500 MB is huge. XI will not be able to handle such a huge payload for sure.
Are you using XI as a mere FTP or are you using Content Conversion with mapping etc?
1. Either use a splitting logic to split the file outside XI ( using Scripts ) and then XI handles these file.
2. or Quick Size your hardware ( Java Heap etc ) to make sure that XI can handle this file ( not recommended though). SAP recommends a size of 5 MBas the optimum size.
Regards
Bhavesh

Popup List with multiple columns but returning a single column (key value)

Hi,
1. Is it not possible to show a POPup (returning a key value) or Select list with more than 1 column? If so how?
2. I have an item in a page which should be hidden if 1 particular button was pressed. This is is to use the same page but have some behavioural change based on which button (option) was selected. How can I get the name of the BUTTON PRESSED TO INVOKE THIS PAGE.
thanks
SebJos

Hi,
In that case, you should make the page item conditional. There are many techniques to do this but it would depend on what the button is for.
As a simple example...
1 - Create a hidden item on your page called, say, Pn_SHOW (replace n with your page number - eg, P1_SHOW). Set the default value of this item to either 0 (zero) or 1 - 0 will hide the item and 1 will display it, so in your example, set it to 0
2 - On the page item you want to show/hide - on its definition, you have a Conditions section. Set the following:
Condition Type: Value of Item in Expression 1 = Expression 2
Expression 1: P1_SHOW
Expression 2: 1
When the page is loaded, because the value of P1_SHOW is 0, and due to the conditions we have just set (where P1_SHOW must be 1 for the item to be shown), the item will not appear on the page.
When you created your button, a new page branch should have been created. On this branch, you have two settings - "Set these items" and "With these values". Set the following:
Set these items: P1_SHOW
With these values: 0
You should also have at least one other branch on your page - one of these will need to set the value of P1_SHOW to 1 to get the item to display.
Andy

Performance of large tables with ADT columns

Hello,
We are planning to build a large table ( 1 billion+ rows) with one of the columns being an Advanced Data Type (ADT) column. The ADT column will be based on a TYPE that will have approximately (250 attributes).
We are using Oracle 10g R2
Can you please tell me the following:
1. How will Oracle store the data in the ADT column?
2. Will the entire ADT record fit in one block?
3. Is it still possible to partition a table on an attribute that is part of the ADT?
4. How will the performace be affected if Oracle does a full table scan of such a table?
5. How much space will Oracle take, if any, for storing a NULL in an ADT?
I think we can create indexes on the attribute of the ADT column. Please let me know if this is not true.
Thanks for your help.

I agree with D.Morgan that object type with 250 attributes is doubtful.
I don't like object tables (tables with "row objects") too.
But, your table is a relational table with object column ("column object").
C.J.Date in An introduction to Database Systems (2004, page 885) says:
"... object/relational systems ... are, or should be, basically just relational systems
that support the relational domain concept (i.e., types) properly - in other words, true relational systems,
meaning in particular systems that allow users to define their own types."
1. How will Oracle store the data in the ADT column?...
For some answers see:
“OR(DBMS) or R(DBMS), That is the Question”
http://www.quest-pipelines.com/pipelines/plsql/tips.htm#OCTOBER
and (of course):
"Oracle® Database Application Developer's Guide - Object-Relational Features" 10g Release 2 (10.2)
http://download-uk.oracle.com/docs/cd/B19306_01/appdev.102/b14260/adobjadv.htm#i1006903
Regards,
Zlatko

Implementing dependent choice list with internal columns as varchar

Hi,
While implementing dependent choice list I created two tables & their views with internal column as varchar(for example: countryName for country table & stateName for states table) instead of int. I couldn't get the DCL working: selecting a value from the dropdown of parent field won't show up only the dependent values of the child field; it will display the dropdown with all values (it will show all values of the States irrespective of the countryname selection). However when I select the internal column as int & visible column as varchar the DCL works fine.
Our requirement is to use the internal columns as varchar instead of int. Please suggest/help.
Thanks,
Madhur

Hi Shashwat,
I have done that. The problem is: dependent choice list doesn't seem work is you choose primarykey/foreign key as varchar. It works if I choose the primarykey/foreign key as int. And this key will be the internal column in the views. My requirement is to have them as varchar so that while checking-in content using RIDC API I don't have to provide the int value.
Thanks & Regards,
Madhur

PIVOT with multiple columns to add and multiple levels of grouping

Hi friends,
I got a table with the columns in the form of:
CRITERIA_A,
CRITERIA_B,
CRITERIA_C,
AMOUNT_A,
AMOUNT_B,
AMOUNT_C,
AMOUNT_D
Any way to design a pivot to present the table reflecting:
Sums of Amount_A, Amount_B, Amount_C, Amount_D
For Rows reflecting grouping levels on:
Criteria_B, Criteria_C
and Columns breakup for:
Criteria_A?
An example like:
Criteria_A1
Criteria_A2
Criteria_A3
Criteria_A4
Row Labels
Sum of Amount_A
Sum of Amount_B
Sum of Amount_C
Sum of Amount_D
Sum of Amount_A
Sum of Amount_B
Sum of Amount_C
Sum of Amount_D
Sum of Amount_A
Sum of Amount_B
Sum of Amount_C
Sum of Amount_D
Sum of Amount_A
Criteria_B3
94
107
36
127
84
132
41
176
24
16
67
29
38
Criteria_C1
24
25
5
49
14
66
5
49
24
16
67
29
Criteria_C2
70
82
31
78
38
Criteria_C3
38
41
31
78
Criteria_C4
32
25
5
49
Criteria_B1
56
142
78
26
32
25
67
8
24
Criteria_C2
24
66
26
8
32
25
67
8
Criteria_C3
32
76
52
18
Criteria_C4
24
Criteria_B2
162
309
264
81
132
230
155
124
14
25
52
8
38
Criteria_C1
38
76
26
8
62
98
98
66
14
25
52
8
38
Criteria_C2
86
157
186
44
70
132
57
58
Criteria_C3
38
76
52
29
Criteria_B4
100
148
130
116
56
142
93
57
46
Criteria_C1
24
66
67
49
32
Criteria_C2
32
76
26
8
14
Criteria_C3
76
82
78
67
Criteria_C4
24
66
52
49
Grand Total
312
558
378
234
348
535
393
424
94
183
212
94
146
Thanx in advance, Best Regards, Faraz A Qureshi

Sounds like this to me if in T-SQL query
(best guess based on what you posted)
SELECT
CRITERIA_B,
CRITERIA_C,
SUM(CASE WHEN CRITERIA_A = 'Critera_A1' THEN AMOUNT_A ELSE 0 END) AS CRITERIA_A1_AMOUNT_A,
SUM(CASE WHEN CRITERIA_A = 'Critera_A1' THEN AMOUNT_B ELSE 0 END) AS CRITERIA_A1_AMOUNT_B,
SUM(CASE WHEN CRITERIA_A = 'Critera_A1' THEN AMOUNT_C ELSE 0 END) AS CRITERIA_A1_AMOUNT_C,
SUM(CASE WHEN CRITERIA_A = 'Critera_A2' THEN AMOUNT_A ELSE 0 END) AS CRITERIA_A2_AMOUNT_A,
SUM(CASE WHEN CRITERIA_A = 'Critera_A2' THEN AMOUNT_B ELSE 0 END) AS CRITERIA_A2_AMOUNT_B,
SUM(CASE WHEN CRITERIA_A = 'Critera_A2' THEN AMOUNT_C ELSE 0 END) AS CRITERIA_A2_AMOUNT_C,
SUM(CASE WHEN CRITERIA_A = 'Critera_AN' THEN AMOUNT_A ELSE 0 END) AS CRITERIA_AN_AMOUNT_A,
SUM(CASE WHEN CRITERIA_A = 'Critera_AN' THEN AMOUNT_B ELSE 0 END) AS CRITERIA_AN_AMOUNT_B,
SUM(CASE WHEN CRITERIA_A = 'Critera_AN' THEN AMOUNT_C ELSE 0 END) AS CRITERIA_AN_AMOUNT_C,
FROM Table
GROUP BY CRITERIA_B,
CRITERIA_C
to make it dynamic see
http://beyondrelational.com/modules/2/blogs/70/posts/10791/dynamic-crosstab-with-multiple-pivot-columns.aspx
However it would be much easier to build this in SSRS reports using matrix if you can use it.
In that case just bring data as is
then add a matrix to report
Add column grouping on Criteria A and Row grouping on Criteria B and then Criteria C .
In data portion add three columns with expressions as
=SUM(Fields!Amount_A.Value)
=SUM(Fields!Amount_B.Value)
=SUM(Fields!Amount_C.Value)
then you will get exact format what you're asking for
Please Mark This As Answer if it solved your issue
Please Vote This As Helpful if it helps to solve your issue
Visakh
My Wiki User Page
My MSDN Page
My Personal Blog
My Facebook Page

Site lookup columns that contain a list with lookup columns help needed

Here is the theoretical scenario,
I have a list called Shirts. The list contains name,color, and size columns.
I have a list called Pants This list contains name, color and size columns as well.
These two lists are contained at the root level of the site. I created site lookup columns for these two lists and used the name, color and size columns as additional fields on both lookup columns. Lets call these lkupPants and lkupShirts.
On a sub site I have a list called people. It contains name,date,lkupPants and lkupShirts columns.
The list looks something like this is dataview:
Name Date lkupPantsName lkupPantsColor lkupPantsSize lkupShirtName lkupShirtColor lkupShirtSize
This is the behavior I expected. The additional columns selected in the site lookup columns appear in the list.
I now create another site lookup column this one is based on the list called people at this subsite level. When creating the site lookup column, the data from lkupPants and lkupShirts are not available.
How can one create a lookup column that contains list data and any lookup columns contained in the list as additional fields.
If anyone can see where I am going with this, can they provide another means to accomplish what I am trying to do?
fr0stsp1re

create additional columns in people list. look up values are available in the workflow so populate the additional columns with the values you need. this will allow you to create another look up column based on this list and all the fields needed.
just a thought.
kashif

SharePoint 2013 WCF Reading a list with a column to allow multiple lookup values

Hi Team,
I'm using SP2013 default WCF in _vti/bin/ListData.svc to read a list.
This list has a look up column and it allows multiple values. While I read the list, I'm not able to retrieve this column as list.
For Example, my parent list is 'VehicleList' and this has a column called-ColorLookUp (lookup with allow multiple values) linked to the 'ColorsList'.
Pseudo Code:
var vehicleItemsList=spcontext.VehicleList.toList();
foreach(var item in vehicleItemsList)
var colorItemList=item.ColorLookUp
I'm expecting the 'colorItemList' should have values but it always gives me the count=0. But the same lookup column works if I dont allow multiple values.
Please help me on this.
thanks

Hi,
For the Lookup column, “Expand” is required if you want to get it through ListData.svc.
A code snippet for your reference:
var items = dc.List1.Expand("MyLookupColumn1");
foreach (var item in items)
Console.WriteLine(item.Title);
foreach (var lk in item.MyLookupColumn1)
Console.WriteLine(lk.Title + "---" + lk.Path);
Feel free to reply if there still any questions.
Thanks
Patrick Liang
Forum Support
Please remember to mark the replies as answers if they
help and unmark them if they provide no help. If you have feedback for TechNet
Subscriber Support, contact [email protected]
Patrick Liang
TechNet Community Support

Best way to handle large list of results in recordsets?

Hello all.
I'm using Dreamweaver CS3, MySQL and ASP/VBScript.
My database of users behind my website is now approaching 25,000.
I often have to "move" items in the database from one user record to another.
Up and until now, I've done this simply by way of a drop down menu/list that is populated with the user ID# and Name of each and every user in the database. This previously allowed me to simply select the ID of the Customer I wanted to "Move" the record to.
The problem, is that the system is of course now trying to load a list of almost 25,000 user ID's each time I view the relevant site page, which is now taking so long to load it's uncomfortable.
I've seen other sites that allow you to start typing something in to a text box and it starts filtering the results that match as you type, showing a list below.
I assume (but am happy to be advised otherwise) that this is likely to be my best way forward, but I haven't the first clue how to do it.
Can anyone advise?
Regards
David.

You're looking for a 'type ahead' control. Try searching the web, although you may have trouble finding example code for classic asp. I did find some asp.net solutions out there.

Handling Large list with 150+ columns(40+ Searchable) and 100000 rows.

Similar Messages

Maybe you are looking for