RE: (forte-users) Optimal number of records to fetch fromForte Cursor

Question

Guys,
The behavior (1 fetch of 20000 vs 2 fetches of 10000 each) may also be DBMS
related. There is potentially high overhead in opening a cursor and initially
fetching the result table. I know this covers a great deal DBMS technology
territory here but one explanation is that the same physical pages may have to
be read twice when performing the query in 2 fetches as compared to doing it in
one shot. Physical IO is perhaps the most expensive (vis a vis- resources)
part of a query. Just a thought.
"Rottier, Pascal" <[email protected]> on 11/15/99 01:34:22 PM
To: "'Forte Users'" <[email protected]>
cc: (bcc: Charlie Shell/Bsg/MetLife/US)
Subject: RE: (forte-users) Optimal number of records to fetch from Forte C
ursor
The reason why a single fetch of 20.000 records performs less then
2 fetches of 10.000 might be related to memory behaviour. Do you
keep the first 10.000 records in memory when you fetch the next
10.000? If not, then a single fetch of 20.000 records requires more
memory then 2 fetches of 10.000. You might have some extra over-
head of Forte requesting additional memory from the OS, garbage
collections just before every request for memory and maybe even
the OS swapping some memory pages to disk.
This behaviour can be controlled by modifying the Minimum memory
and Maximum memory of the partition, as well as the memory chunk
size Forte uses to increment its memory.
Upon partition startup, Forte requests the Minimum memory from the
OS. Whithin this area, the actual memory being used grows, until
it hits the ceiling of this space. This is when the garbage collector
kicks in and removes all unreferenced objects. If this does not suffice
to store the additional data, Forte requests 1 additional chunk of a
predefined size. Now, the same behaviour is repeated in this, slightly
larger piece of memory. Actual memory keeps growing until it hits
the ceiling, upon which the garbage collector removes all unrefer-
enced objects. If the garbage collector reduces the amount of
memory being used to below the original Miminum memory, Forte
will NOT return the additional chunk of memory to the OS. If the
garbage collector fails to free enough memory to store the new data,
Forte will request an additional chunk of memory. This process is
repeated untill the Maximum memory is reached. If the garbage
collector fails to free enough memory at this point, the process
terminates gracelessly (which is what happens sooner or later when
you have a memory leak; something most Forte developpers have
seen once or twice).
Pascal Rottier
STP - MSS Support & Coordination Group
Philip Morris Europe
e-mail: [email protected]
Phone: [phone]
+++++++++++++++++++++++++++++++++++
Origin IT-services
Desktop Business Solutions Rotterdam
e-mail: [email protected]
Phone: [phone]
+++++++++++++++++++++++++++++++++++
/* All generalizations are false! */
-----Original Message-----
From: [email protected] [SMTP:[email protected]]
Sent: Monday, November 15, 1999 6:53 PM
To: [email protected]
Subject: (forte-users) Optimal number of records to fetch from Forte
Cursor
Hello everybody:
I 'd like to ask a very important question.
I opened Forte cursor with approx 1.2 million records, and now I am trying
to figure out the number of records per fetch to obtain
the acceptable performance.
To my surprise, fetching 100 records at once gave me approx 15 percent
performance gain only in comparsion
with fetching records each by each.
I haven't found significant difference in performance fetching 100, 500
or
10.000 records at once.In the same time, fetching 20.000
records at once make a performance approx 20% worse( this fact I cannot
explain).
Does anybody have any experience in how to improve performance fetching
from
Forte cursor with big number of rows ?
Thank you in advance
Genady Yoffe
Software Engineer
Descartes Systems Group Inc
Waterloo On
Canada
For the archives, go to: http://lists.sageit.com/forte-users and use
the login: forte and the password: archive. To unsubscribe, send in a new
email the word: 'Unsubscribe' to: [email protected]
For the archives, go to: http://lists.sageit.com/forte-users and use
the login: forte and the password: archive. To unsubscribe, send in a new
email the word: 'Unsubscribe' to: [email protected]

Answer 1

Hi Kieran,
According to your description, you are going to figure out what is the optimal number of records per partition, right? As per my understanding, this number was change by your hardware. The better hardware you have, the more number of records per partition.
The earlier version of the performance guide for SQL Server 2005 Analysis Services Performance Guide stated this:
"In general, the number of records per partition should not exceed 20 million. In addition, the size of a partition should not exceed 250 MB."
Besides, the number of records is not the primary concern here. Rather, the main criterion is manageability and processing performance. Partitions can be processed in parallel, so the more there are the more can be processed at once. However, the more partitions
you have the more things you have to manage. Here is some links which describe the partition optimization
http://blogs.msdn.com/b/sqlcat/archive/2009/03/13/analysis-services-partition-size.aspx
http://www.informit.com/articles/article.aspx?p=1554201&seqNum=2
Regards,
Charlie Liao
TechNet Community Support