Dealing with large volumes of data

Background:
I recently "inherited" support for our company's "data mining" group, which amounts to a number of semi-technical people who have received introductory level training in writing SQL queries and been turned loose with SQL Server Management
Studio to develop and run queries to "mine" several databases that have been created for their use. The database design (if you can call it that) is absolutely horrible. All of the data, which we receive at defined intervals from our
clients, is typically dumped into a single table consisting of 200+ varchar(x) fields. There are no indexes or primary keys on the tables in these databases, and the tables in each database contain several hundred million rows (for example one table
contains 650 million rows of data and takes up a little over 1 TB of disk space, and we receive weekly feeds from our client which adds another 300,000 rows of data).
Needless to say, query performance is terrible, since every query ends up being a table scan of 650 million rows of data. I have been asked to "fix" the problems.
My experience is primarily in applications development. I know enough about SQL Server to perform some basic performance tuning and write reasonably efficient queries; however, I'm not accustomed to having to completely overhaul such a poor design
with such a large volume of data. We have already tried to add an identity column and set it up as a primary key, but the server ran out of disk space while trying to implement the change.
I'm looking for any recommendations on how best to implement changes to the table(s) housing such a large volume of data. In the short term, I'm going to need to be able to perform a certain amount of data analysis so I can determine the proper data
types for fields (and whether any existing data would cause a problem when trying to convert the data to the new data type), so I'll need to know what can be done to make it possible to perform such analysis without the process consuming entire days to analyze
the data in one or two fields.
I'm looking for reference materials / information on how to deal with the issues, particularly when a large volumn of data is involved. I'm also looking for information on how to load large volumes of data to the database (current processing of a typical
data file takes 10-12 hours to load 300,000 records). Any guidance that can be provided is appreciated. If more specific information is needed, I'll be happy to try to answer any questions you might have about my situation.

I don't think you will find a single magic bullet to solve all the issues. The main point is that there will be no shortcut for major schema and index changes. You will need at least 120% free space to create a clustered index and facilitate
major schema changes.
I suggest an incremental approach to address you biggest pain points. You mention it takes 10-12 hours to load 300,000 rows, which suggests there may be queries involved in the process which require full scans of the 650 million row table. Perhaps
some indexes targeted at improving that process is a good first step.
What SQL Server version and edition are you using? You'll have more options with Enterprise (partitioning, row/page compression).
Regarding the data types, I would take a best guess at the proper types and run a query with TRY_CONVERT (assuming SQL 2012) to determine counts of rows that conform or not for each column. Then create a new table (using SELECT INTO) that has strongly
typed columns for those columns that are not problematic, plus the others that cannot easily be converted, and then drop the old table and rename the new one. You can follow up later to address columns data corrections and/or transformations.
Dan Guzman, SQL Server MVP, http://www.dbdelta.com

Similar Messages

Dealing with large amounts of data

Hi
I am new to using Flex and BlazeDS. I can see in the FAQ that binary data transfer from server to Flex app is more efficient. My question is: is there a way to build a Flex databound control (e.g. datagrid) which binds to a SQL query or web service or remoting on the server side and then display infinite amount of data as user pages or scrolls using scrollbar? or does the developer have to write code from scratch to deal with paging or deal with infinite scrollbar by asking server for chunks of data at a time?

You have to write your own paginating grid. It's easy to do, just make a canvas, throw a grid and some buttons on it, than when user click to the next page, you make a request to the server and when you have a result, set it as a new data model for the grid.
I would discourage you to return infinite amount of data to the user, provide a search functionality plus pagination.
Hope that helps.

How to deal with large amount of data

Hi folks,
I have a web page that needs 3 maps to serve. Each map will hold about 150,000 entries, and all of them will use about 100MB total. For some users with lots of data (e.g. 1,000,0000 entries), it may use up to 1GB of memory. If few of these high-volume users log in at the same time, it can bring the server down. The data is from the files, I cannot read it on demand because it will be too slow. Loading the data to maps offers me very good performance, but it does not scale. I am thinking to serialize the maps and deserialize them when I need. Is it my only option?
Thanks in advance!

JoachimSauer wrote:
I don't buy the "too slow" argument.
I'd put the data into a database. They are built to handle that amount of data.++

Processing large volumes of data in PL/SQL

I'm working on a project which requires us to process large volumes of data on a weekly/monthly/quarterly basis, and I'm not sure we are doing it right, so any tips would be greatly appreciated.
Requirement
Source data is in a flat file in "short-fat" format i.e. each data record (a "case") has a key and up to 2000 variable values.
A typical weekly file would have maybe 10,000 such cases i.e. around 20 million variable values.
But we don't know which variables are used each week until we get the file, or where they are in the file records (this is determined via a set of meta-data definitions that the user selects at runtime). This makes identifying and validating each variable value a little more interesting.
Target is a "long-thin" table i.e. one record for each variable value (with numeric IDs as FKs to identify the parent variable and case.
We only want to load variable values for cases which are entirely valid. This may be a merge i.e. variable values may already exist in the target table.
There are various rules for validating the data against pre-existing data etc. These rules are specific to each variable, and have to be applied before we put the data in the target table. The users want to see the validation results - and may choose to bail out - before the data is written to the target table.
Restrictions
We have very limited permission to perform DDL e.g. to create new tables/indexes etc.
We have no permission to use e.g. Oracle external tables, Oracle directories etc.
We are working with standard Oracle tools i.e. PL/SQL and no DWH tools.
DBAs are extremely resistant to giving us more disk space.
We are on Oracle 9iR2, with no immediate prospect of moving to 10g.
Current approach
Source data is uploaded via SQL*Loader into static "short fat" tables.
Some initial key validation is performed on these records.
Dynamic SQL (plus BULK COLLECT etc) is used to pivot the short-fat data into an intermediate long-thin table, performing the validation on the fly via a combination of including reference values in the dynamic SQL and calling PL/SQL functions inside the dynamic SQL. This means we can pivot+validate the data in one step, and don't have to update the data with its validation status after we've pivoted it.
This upload+pivot+validate step takes about 1 hour 15 minutes for around 15 million variable values.
The subsequent "load to target table" step also has to apply substitution rules for certain "special values" or NULLs.
We do this by BULK collecting the variable values from the intermediate long-thin table, for each valid case in turn, applying the substitution rules within the SQL, and inserting into/updating the target table as appropriate.
Initially we did this via a SQL MERGE, but this was actually slower than doing an explicit check for existence and switching between INSERT and UPDATE accordingly (yes, that sounds fishy to me too).
This "load" process takes around 90 minutes for the same 15 million variable values.
Questions
Why is it so slow? Our DBAs assure us we have lots of table-space etc, and that the server is plenty powerful enough.
Any suggestions as to a better approach, given the restrictions we are working under?
We've looked at Tom Kyte's stuff about creating temporary tables via CTAS, but we have had serious problems with dynamic SQL on this project, so we are very reluctant to introduce more of it unless it's absolutely necessary. In any case, we have serious problems getting permissions to create DB objects - tables, indexes etc - dynamically.
So any advice would be gratefully received!
Thanks,
Chris

We have 8 "short-fat" tables to hold the source data uploaded from the source file via SQL*Loader (the SQL*Loader step is fast). The data consists simply of strings of characters, which we treat simply as VARCHAR2 for the most part.
These tables consist essentially of a case key (composite key initially) plus up to 250 data columns. 8*250 = 2000, so we can handle up to 2000 of these variable values. The source data may have 100 any number of variable values in each record, but each record in a given file has the same structure. Each file-load event may have a different set of variables in different locations, so we have to map the short-fat columns COL001 etc to the corresponding variable definition (for validation etc) at runtime.
CASE_ID VARCHAR2(13)
COL001 VARCHAR2(10)
COL250 VARCHAR2(10)
We do a bit of initial validation in the short-fat tables, setting a surrogate key for each case etc (this is fast), then we pivot+validate this short-fat data column-by-column into a "long-thin" intermediate table, as this is the target format and we need to store the validation results anyway.
The intermediate table looks similar to this:
CASE_NUM_ID NUMBER(10) -- surrogate key to identify the parent case more easily
VARIABLE_ID NUMBER(10) -- PK of variable definition used for validation and in target table
VARIABLE_VALUE VARCHAR2(10) -- from COL001 etc
STATUS VARCHAR2(10) -- set during the pivot+validate process above
The target table looks very similar, but holds cumulative data for many weeks etc:
CASE_NUM_ID NUMBER(10) -- surrogate key to identify the parent case more easily
VARIABLE_ID NUMBER(10) -- PK of variable definition used for validation and in target table
VARIABLE_VALUE VARCHAR2(10)
We only ever load valid data into the target table.
Chris

How to get around a performance issue when dealing with a lot of data

Hello All,
This is an academic question really, I'm not sure what I'm going to do with my issue, but I have some options. I was wondering if anyone would like to throw in their two cents on what they would do.
I have a report, the users want to see all agreements and all conditions related to the updating of rebates and the affected invoices. From a technical perspective ENT6038-KONV-KONP-KONA-KNA1. THese are the tables I have to hit. The problem is that when they retroactively update rebate conditions they can hit thousands of invoices, which blossoms out to thousands of conditions...you see the problem. I simply have too much data to grab, it times out.
I've tried everything around the code. If you have a better way to get price conditions and agreement numbers off of thousands of invoices, please let me know what that is.
I have a couple of options.
1) Use shared memory to preload the data for the report. This would work, but I'm not going to know what data is needed to be loaded until report run time. They put in a date. I simply can't preload everything. I don't like this option much.
2) Write a function module to do this work. When the user clicks on the button to get this particular data, it will launch the FM in background and e-mail them the results. As you know, the background job won't time out. So far this is my favored option.
Any other ideas?
Oh...nope, BI is not an option, we don't have it. I know, I'm not happy about it. We do have a data warehouse, but the prospect of working with that group makes me whince.

My two cents - firstly totally agree with Derick that its probably a good idea to go back to the business and justify their requirement in regards to reporting and "whether any user can meaningfully process all those results in an aggregate". But having dealt with customers across industries over a long period of time, it would probably be bit fanciful to expect them to change their requirements too much as in my experience neither do they understand (too much) technology nor they want to hear about technical limitations for a system etc. They want what they want if possible yesterday!
So, about dealing with performance issues within ABAP, I'm sure you must be already using efficient programming techniques like using Hash internal tables with Unique Keys, accessing rows of the table using Field-Symbols and all that but what I was going to suggest to you is probably look at using [Extracts|http://help.sap.com/saphelp_nw04/helpdata/en/9f/db9ed135c111d1829f0000e829fbfe/content.htm]. I've had to deal with this couple of times in the past when dealing with massive amount of data and I found it to be very efficient in regards to performance. A good point to remember when using Extracts that, I quote from SAP Help, "The size of an extract dataset is, in principle, unlimited. Extracts larger than 500KB are stored in operating system files. The practical size of an extract is up to 2GB, as long as there is enough space in the filesystem."
Hope this helps,
Cheers,
Sougata.

Retrive SQL from Webi report and process it for large volume of data

We have a Scenario where, we need to extract large volumes of data into flat files and distribute them from u2018Teradatau2019 warehouse which we usually call them as u2018Extractsu2019. But the requirement is such that, Business users wants to build their own u2018Adhoc Extractsu2019. The only way, I thought, to achieve this, is Build a universe, create the query, save the report and do not run it. Then write a RAS SDK to retrieve the SQL code from the reports and save into .txt file and process it directly in Teradata?
Is there any predefined Solution available with SAP BO or any other tool for this kind of Scenarios?

Hi Shawn,
Do we have some VB macro to retrieve Sql queries of data providers of all the WebI reports in CMS.
Any information or even direction where I can get information will be helpful.
Thanks in advance.
Ashesh

Can ui:table deal with large table?

I have found h:dataTable can do pagination because it's data source is just a DataModel. But ui:table's datasouce is a data provider which looks some complex and confused.
I have a large table and I want to load the data on-demand . So I try to implement a provider. But soon I found that the ui:table may be load all data from provider always.
In TableRowGroup.java, there are many code such as:
provider.getRowKeys(rowCount, null);
null will let provider load all data.
So ui:table can NOT deal with large table!?
thx.
fan

But ui:table just uses TableDataProvider interface.TableData provider is a wrapper for the CachedRowSet
There are two layers between the UI:Table comonent and the database table: the RowSet layer and the Data Provider layer. The RowSet layer makes the connection to the database, executes the queries, and manages the result set. The Data Provider layer provides a common interface for accessing many types of data, from rowsets, to Array objects, to Enterprise JavaBeans objects.
Typically, the only time that you work with the RowSet object is when you need to set query parameters. In most other cases, you should use the Data Provider to access and manipulate the data.
What can a CachedRowSet (or CachedRowSetprovider?)
do?Check out the API that I pointed you to to see what you can do with a CachedRowSet
Does the Table cache the data itself?
Maybe this way is convenient for filter and order?
Thx.I do not know the answers to these questions.

XSU: Dealing with large tables / large XML files

Hi,
I'm trying to generate a XML file from a "large" table (about 7 million lines, 512Mbytes of storage) by means of XSU. I get into "java.lang.OutOfMemoryError" even after raising the heap size up to 1 Gbyte (option -Xmx1024m of the java cmd line).
For the moment, I'm involved in an evaluation process. But in a near future, our applications are likely to deal with large amount of XML data, (typically hundreds of Mbytes of storage, which means possibly Gbytes of XML code), both in updating/inserting data and producing XML streams from existing data in relationnal DB.
Any ideas about memory issues regarding XSU? Should we consider to use XMLType instead of "classical" relational tables loaded/unloaded by means of XSU?
Any hint appreciated.
Regards,
/Hervi QUENIVET
P.S. our environment is Linux red hat 7.3 and Oracle 9.2.0.1 server

Hi,
I'm trying to generate a XML file from a "large" table (about 7 million lines, 512Mbytes of storage) by means of XSU. I get into "java.lang.OutOfMemoryError" even after raising the heap size up to 1 Gbyte (option -Xmx1024m of the java cmd line).
For the moment, I'm involved in an evaluation process. But in a near future, our applications are likely to deal with large amount of XML data, (typically hundreds of Mbytes of storage, which means possibly Gbytes of XML code), both in updating/inserting data and producing XML streams from existing data in relationnal DB.
Any ideas about memory issues regarding XSU? Should we consider to use XMLType instead of "classical" relational tables loaded/unloaded by means of XSU?
Any hint appreciated.
Regards,
/Hervi QUENIVET
P.S. our environment is Linux red hat 7.3 and Oracle 9.2.0.1 server Try to split the XML before you process it. You can take look into XMLDocumentSplitter explained in Building Oracle XML Applications Book By Steven Meunch.
The other alternative is write your own SAX handler and send the chuncks of XML for insert

How do I pause an iCloud restore for app with large amounts of data?

I am using an iPhone app which is holding 10 Gb of data (media files) .
Unfortunately, although all data was backed up, my iPhone 4 was faulty and needed to be replaced with a new handset. On restore, the 10Gb of data takes a very long time to restore over wi-fi. If interrupted (I reached the halfway point during the night) to go to work or take the dog for a walk, I end up of course on 3G for a short period of time.
Next time I am in a wi-fi zone the app is restoring again right from the beginning
How does anyone restore an app with large amounts of data or pause a restore?

You can use classifications but there is no auto feature to archive like that on web apps.
In terms of the blog, Like I have said to everyone that has posted about blog preview images:
http://www.prettypollution.com.au/business-catalyst-blog
Just one example of an image at the start of the blog post rendering out, not hard at all.

What is best way dealing with large tiff file in OSX Lion?

I'm working with large tiff file (engineering drawing), but preview couldnt handle (unresponsive) this situation.
What is best way dealing with large tiff file in OSX Lion? (Viewing only or simple editing)
Thx,
54n9471

Use an iPad and this app http://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=400600005&mt=8

Premiere Pro 2.0 slows when dealing with larger video files

I'm having issues with Premiere Pro 2.0 slowing to a crawl and taking 60-90 seconds to come back to life when dealing with larger .avi's (8+ mins). When I try to play a clip on the timeline, drag the slider over said clip, or play from a title into said clip on the timeline, Premiere hangs. The clips on question are all rendered, and the peak file has been generated for each different clip has well. This is a new problem; the last time I was working with a larger clip (45+ mins, captured from a Hi-8 cam), I had no problems. Now, I experience this slow down with all longer clips, although I've only dealt with footage captured from a Hi-8 cam and also a mini-DV cam. This problem has made Premiere nearly unusable. I'm desperate at this point.
System:
CPU: P4 HT 2.4ghz
Ram: 2x 1gb DDR
Video: ATI Radeon 9000 Series
Scratch Disk: 250gb WD My Book - USB 2.0 (I suspect this might be part of the problem)
OS: XP Pro SP2
I'm not on my machine right now, and I can definitely provide more information if needed.
Thanks in advance.

Aside from some other issues, I found that USB was just not suited for editing to/from, and on a much faster machine, that you list.
FW-400 was only slightly better. It took FW-800, before I could actually use the externals for anything more than storage, i.e. no editing, just archiving.
eSATA would be even better/faster.
Please see Harm's ARTICLES on hardware, before you begin investing.
Good luck,
Hunt
[Edit] Oops, I see that Harm DID link to his articles. Missed that. Still, it is worth mentioning again.
Also, as an aside, PrPro 2.0 has no problem on my workstation when working with several 2 hour DV-AVI's, even when these are edited to/from FW-800 externals.
Message was edited by: the_wine_snob - [Edit]

In OSB , xquery issue with large volume data

Hi ,
I am facing one problem in xquery transformation in OSB.
There is one xquery transformation where I am comparing all the records and if there are similar records i am clubbing them under same first node.
Here i am reading the input file from the ftp process. This is perfectly working for the small size input data. When there is large input data then also its working , but its taking huge amount of time and the file is moving to error directory and i see the duplicate records created for the same input data. I am not seeing anything in the error log or normal log related to this file.
How to check what is exactly causing the issue here, why it is moving to error directory and why i am getting duplicate data for large input( approx 1GB).
My Xquery is something like below.
<InputParameters>
                for $choice in $inputParameters1/choice
                 let $withSamePrimaryID := ($inputParameters1/choice[PRIMARYID eq $choice/PRIMARYID])
                 let $withSamePrimaryID8 := ($inputParameters1/choice[FIRSTNAME eq $choice/FIRSTNAME])
                 return
                  <choice>
                 if(data($withSamePrimaryID[1]/ClaimID) = data($withSamePrimaryID8[1]/ClaimID)) then
                 let $claimID:= $withSamePrimaryID[1]/ClaimID
                 return
                 <ClaimID>{$claimID}</ClaimID>
                 else
                 <ClaimID>{ data($choice/ClaimID) }</ClaimID>

HI ,
I understand your use case is
a) read the file ( from ftp location.. txt file hopefully)
b) process the file ( your x query .. although will not get into details)
c) what to do with the file ( send it backend system via Business Service?)
Also noted the files with large size take long time to be processed . This depends on the memory/heap assigned to your JVM.
Can say that is expected behaviour.
the other point of file being moved to error dir etc - this could be the error handler doing the job ( if you one)
if no error handlers - look at the timeout and error condition scenarios on your service.
HTH

What is the best way to extract large volume of data from a BW InfoCube?

Hello experts,
Wondering if someone can suggest the best method that is availabe in SAP BI 7.0 to extract a large amount of data (approx 70 million records) from an InfoCube. I've tried OpenHub and APD but not working. I always need to separate the extracts into small datasets. Any advice is greatly appreciated.
Thanks,
David

Hi David,
We had the same issue but that was loading from an ODS to cube. We have over 50 million records. I think there is no such option like parallel loading using DTPs. As suggested earlier in the forum, the only best option is to split according to the calender year of fis yr.
But remember even with the above criteria sometimes for some cal yr you might have lot of data, even that becomes a problem.
What i can suggest you is apart from Just the cal yr/fisc, also include some other selection criteria like comp code or sales org.
yes you will end up load more requests, but the data loads would go smooth with lesser volumes.
Regards
BN

Dealing with large files, again

Ok, so I've looked into using BufferedReaders and can't get my head round them; or more specifically, I can't work out how to apply them to my code.
I have inserted a section of my code below, and want to change it so that I can read in large files (of over 5 million lines of text). I am reading the data into different arrays and then processing them. Obvioulsy, when reading in such large files, my arrays are filling up and failing.
Can anyone suggest how to read the file into a buffer, deal with a set amount of data, process it, empty the arrays, then read in the next lot?
Any ideas?
void readV2(){
     String line;
     int i=0,lineNo=0;
        try {
          //Create input stream
          FileReader fr = new FileReader(inputFile);
             BufferedReader buff = new BufferedReader(fr);
            while((line = buff.readLine()) != null) {
          if(line.substring(0,2).equals("V2")){
                 lineNo = lineNo+1;
          IL[i] = Integer.parseInt(line.substring(8,15).trim());
                //Other processing here
                 NoOfPairs = NoOfPairs+1;
                 }//end if
                 else{
                  break;
        }//end while
        buff.close();
        fr.close();
        }//end try
        catch (IOException e) {
        log.append("IOException error in readESSOV2XY" + e + newline);
        proceed=false;
        }//end catch IOException
        catch (ArrayIndexOutOfBoundsException e) {
               arrayIndexOutOfBoundsError(lineNo);
     }//end catch ArrayIndexOutOfBoundsException
     catch (StringIndexOutOfBoundsException e) {
          stringIndexOutOfBoundsError(e.getMessage(),lineNo);
}//end V2Many thanks for any help!
Tim

Yeah, ok, so that seems simple enough.
But once I have read part of the file into my program,
I need to call another method to deal with the data I
have read in and write it out to an output file.
How do I get my file reader to "remember" where I am
up to in the file I'm reading?
An obvious way, but possibly not too good technically,
would be to set a counter and when I go back to the
fiel reader, skip that number of lines in the inpuit
file.
This just doesn't seem too efficient, which is
critical when it comes to dealing with such large
files (i.e. several million lines long)I think you might need to change the way you are thinking about streams. The objective of a stream is to read and process data at the same time.
I would recommend that you re-think your algorithm : instead of reading the whole file, then doing your processing - think about how you could read a line and process a line, then read the next line, etc...
By working on just the pieces of data that you have just read, you can process huge files with almost no memory requirements.
As a rule of thumb, if you ever find yourself creating huge arrays to hold data from a file, chances are pretty good that there is a better way. Sometimes you need to buffer things, but very rarely do you need to buffer such huge pieces.
- K

Tips for dealing with large channel count on cRio

Hello, I have a very simple application that takes an analog input using an AI module (9205) from a thermistor and based on the value of the input it sends out a true/false signal using a digital out module (9477). Each cRio chassis will have close to 128 channels, the code being exactly the same for each channel.
I wonder if anyone has any tips for how I can do this so that I don't have to copy and paste each section of code 128 times. Obviously this would be a nightmare if the code ever had to be changed. I'm sure there is a way to make a function or a class but being new to graphical programming I can't think of a good way to do this. I looked for a way to dynamically select a channel but can't seem to find anything, if I could select the channel dynamically I'm guessing I can create a subvi and do it that way. Any tips or help would be greatly appreciated.

There isn't a way to dynamically choose a channel at runtime. In order
for the VI to compile successfully, the compiler must be able to statically
determine which channel is being read or written in the I/O Node at
compile time. However, that doesn't mean you can't write a reusable
subvi. If you right click on the FPGA I/O In terminal of the I/O Node and create a constant or control, you should be able to reuse the same logic for all of your channels. The attached screen shot should illustrate the basics of what this might look like. If you right click the I/O control/constant and select "Configure I/O Type...", you can configure the interface the I/O Item must support in order for it to be selectable from the control. While this helps single source some of the logic, you will still eventually need 128 I/O constants somewhere in your FPGA VI hierarchy.
I should also mention that if each channel being read from the 9205 is contained in a separate subVI or I/O Node, you will also incur some execution time overhead due to the scanning nature of the module. You mentioned you are reading temperature signals so the additional execution time may not be that important to you. If it is, you may want to look at the IO Sample Method. You can find more information and examples on how to use this method in the LV help. Using the IO Sample Method does allow you to dynamically choose a channel at runtime and is generally more efficient for high channel counts. However, it's also a lot more complicated to use than the I/O Node.
You also mentioned concerns about the size of arrays and the performance implications of using a single for loop to iterate across your data set. That's the classic design trade off when dealing with FPGAs. If you want to perform as much in parallel as possible, you'll need to store all 128 data points from the 9205 modules at once, process the data in parallel using 128 instances of the same circuit, and then output a digital value based on the result. If you're using fixed point data types, that's 182 x 26 bits for just the I/O data from the 9205. While this will yield the fastest execution times, the resulting VI may be too large to fit on your target. Conversely, you could use the IO Sample Method to read each channel one at a time, process the data using the same circuit, and then output a digital value. This strategy will use the least amount of logic on the FPGA but will also take the longest to execute. Of course, there are all sorts of options you could create in between these two extremes. Without knowing more about your requirements, it's hard to advise which end of the spectrum you should shoot for. Anyway, hopefully this will give you some ideas on where to get started.
Attachments:
IO Constant.JPG ‏31 KB

Dealing with large volumes of data

Similar Messages

Maybe you are looking for