Advice needed: The way to solve out of memory problem (or the way to work with big csv files)
Hello:)
I'm in trouble: I have a big csv file (over 5gb of web-analytics data) and my 64 bit excel (and 6gb ram)
I cant load file to data model because of it's size. There is an error "out of memory" in power query.
This is the first time when I encountered such a problem.
What options do I have to work with such a file? To increase memory in my computer? Would it solve the problem? How much do I need to work with 6gb csv?
Or may be I can upload my data somewhere to azure and work with it there?
So the problem - is there any way to deal with big files using power query? Or I need to become a developer and learn sql or other languages?
Thanks in advance.
Max
Hi Miguel!
Thanks for your answer.
I've tried to load this file on virtual pc from azure cloud with this config:
I have increased memory limit in power query settings:
And still, the proble is the same:
What I do wrong?
Similar Messages
-
Out of memory problem using the API
Hi all,
I need your assistance, we are working with CDB 10.2 making searches and retrieving the documents with all their attributes.
In our actual scenario we have a single user (which represents an application) accessing CDB. This user use several persistent sessions simultaneously. I mean, several thousands of final users connect to an application that uses one user of CDB to connect to CDB with several persistent sessions.
To simulate this scenario we wrote java code that open five threads and make several searches (requesting all the attributes) using the same user on cdb.
Retrieving a considerable amount of data found on the search (~5000), we found a “Out of memory” problem when we made these tests:
- 5 threads obtaining 100 documents (and all their attributes) / search
- 1 thread obtaining 500 documents (and all their attributes) / search
- Also we have same problem if we make several searches with less results
We suppose it’s a configuration or code issue so we ask for your assistance and experience to solve it.
Thanks for your help,
Dani
import java.sql.Connection;
import oracle.ifs.examples.api.constants.AttributeRequests;
import oracle.ifs.examples.api.util.CommonUtils;
import oracle.ifs.fdk.Attributes;
import oracle.ifs.fdk.ClientUtils;
import oracle.ifs.fdk.FdkConstants;
import oracle.ifs.fdk.FdkCredential;
import oracle.ifs.fdk.ManagersFactory;
import oracle.ifs.fdk.NamedValue;
import oracle.ifs.fdk.Options;
import oracle.ifs.fdk.SearchExpression;
import oracle.ifs.fdk.SearchManager;
import oracle.ifs.fdk.SimpleFdkCredential;
import oracle.jdbc.pool.OracleDataSource;
public class Prueba {
public static void main(String args[]) {
Thread thread = new BasicThread1();
Thread thread1 = new BasicThread1();
Thread thread2 = new BasicThread1();
Thread thread3 = new BasicThread1();
Thread thread4 = new BasicThread1();
thread.start();
thread1.start();
thread2.start();
thread3.start();
thread4.start();
class BasicThread1 extends Thread {
public void run() {
ManagersFactory session = null;
try {
System.out.println(this.getName() + "-->init");
session = getSession();
SearchManager sManager = session.getSearchManager();
SearchExpression srchExpr = new SearchExpression(Attributes.SIZE,
new Integer(20000000), FdkConstants.OPERATOR_LESS_THAN);
NamedValue[] res = null;
for (int i = 0; i < 100000; i++) {
res = sManager.search(srchExpr, basicSearchOptions2,
AttributeRequests.DOCUMENT_CATEGORY_ATTRIBUTES);
System.out.println(this.getName()+" --> fin sin error: " + res.length);
} catch (Throwable t) {
t.printStackTrace();
System.out.println("<--"+this.getName());
} finally {
CommonUtils.bestEffortLogout(session);
static NamedValue[] basicSearchOptions2 = new NamedValue[] {
ClientUtils.newNamedValue(
Options.MULTILEVEL_FOLDER_RESTRICTION,
Boolean.TRUE),
ClientUtils.newNamedValue(Options.SEARCH_FOR_DOCUMENTS,
Boolean.TRUE),
ClientUtils.newNamedValue(Options.SEARCH_FOR_FOLDERS,
Boolean.FALSE),
ClientUtils.newNamedValue(Options.RETURN_COUNT,
new Integer(500)) //<<Maximo nº de elementos
private static ManagersFactory getSession() throws Exception {
OracleDataSource ods = new OracleDataSource();
ods.setURL("URL");
Connection conn = ods.getConnection();
FdkCredential credential = new SimpleFdkCredential("USER","PSW");
ManagersFactory session = ManagersFactory.login(credential,
"SERVER");
return session;
}re-Post
-
What´s the best approach to work with Excel, csv files
Hi gurus. I got a question for you. According to your experience what's the best approach to work with Excel or csv files that have to be uploaded through DataServices to you datawarehouse.
Let's say your end-user, who is not a programmer, creates a group of 4 excel files with different calculations in a monthly basis, so they can generate a set of reports from their datawarehouse once the files have been uploaded to tables in your DWH. The calculations vary from month to month. The user doesn't have a front-end to upload the excel files directly to Data Services. The end user needs to keep a track of which person uploaded the files for a determined month.
1. The end user should place their 4 excel files in a shared directory that will be seen by DataServices.
2. DataServices will execute certain scheduled job that will read the four files and upload them to the Datawarehouse at a determined time, lets say at 9:00pm.
It makes me wonder... what happens if the user needs to present their reports immediately so they can´t wait until 9:00pm. Is it possible for the end user to execute some kind of action (out of the DataServices Environment) so DataServices "could know" that it has to process those files right now, instead of waiting for the night schedule?
Is there a way that DS will track who was the person who uploaded those files?
Would it be better to build a front-end for the end user so they can upload their four files directlyto the datawarehouse?
Waiting for your comments to resolve this dilemma
Best Regards
ErikaHi,
There are functions in DS that captures the input files automatically. You could use file_exists() or wait_for_file() option to do that. Schedule the job to run every certain minute and if the file exists then run. This could be done by using a certain file name with date and timestamp etc or after running move the old files to archive and DS wait for new files to show up.
Check this - Selective Reading and Postprocessing - Enterprise Information Management - SCN Wiki
Hope this helps.
Arun -
Why does the iPad 2 run out of memory
Why does the iPad 2 run out of memory?
First ...read deggie's response. Makes sense - even a computer running a large amount of RAM can run short on memory depending upon the operation you are performing so sometimes you have to do what is available to "clear" the memory.
You can ...close apps and restart the iPad on occasion. You close apps by double tapping the home button to bring up your recent apps in the multitasking tray. Tap and hold down on any of the icons until they all begin to wiggle. Then tap the minus signs in the corner of the apps to close them. Close all of your apps in that area. Also take note of how many apps are down there. That may have something to do with why the iPad is running out of memory. Tap the home button twice to return to the home screen. Then restart your iPad by holding down on the sleep button until the red slider appears and slide to shut off. To power up, hold the sleep button until the Apple logo appears and let go of the button.
If you do this once in a while, it may make a difference in how your iPad performs. In any event, it will not hurt anything. -
Flex 4 RichEditableText out of memory problems
Hello, we're conducting performance testing on the UI of a productivity application we're developing using Adobe AIR. We are using Flash Builder 4 public beta. One part of the performance test is updating the textFlow property of a RichEditableText control every 5 seconds with random data that contains several paragraphs and several images (to mimic a typical news article). We used TextFlowUtil.importFromString(str) to convert the raw data to a textFlow object.
We found that the above test with the RichEditableText control would quickly crash the AIR runtime with presumably out-of-memory problems. If we switch to using the (read-only) RichText control, the problem went away. The only material difference between the two controls we could find in the documentation is that the RichEditableText control supports unlimited undo/redos as long as it retains focus. Could this be the source of the memory problem? Regardless, can its behavior be modified to avoid the out-of-memory problems?
The use case we are trying to simulate is having the productivity app open for weeks at a time, with constant editing and going back and forth between different workflows.
Thanks for your help, and please let me know if you need additional informationTry a more recent build.
Alex Harui
Flex SDK Developer
Adobe Systems Inc.
Blog: http://blogs.adobe.com/aharui -
Is there a way to find out details about, who approved the RMA on system using front end or back end?
But is there a way (simple or not-so-simple) to find
out EVERY SINGLE FILE that's referenced on a
timeline? (I have lots of stuff imported into the
project that I know isn't used on any timeline, but
I'm afraid of moving or deleting things for fear of
never finding it again.)
I'd love to get some sort of "printout" (a list or
something) of every single file.
There is, go to browser and cntrl click on any one of the columns,
and select - show/source. A new column named "source" will pop up and will show where everything "lives". Click on the header at the top and it will justify the column as well. You'll notice this won't show anything for your seq.'s. of course, but will show stills/audio/motion files...
Once you have the new column, you can drag it next to the "name" column and get a grab of it and print it if you like, or whatever.
If you want to do this for just 1 seq., then can create a new bin and double click it to open it. Cntrl-A (select all) in your main seq. and just drag all into the new bin. You will see only what's in that seq. with the "source" column right next to it.
From what I've read, you seem to get the MM thing and it's obviously what you'll need to do to archive as the guys said. But what your missing is a file path system to where everything resides.
Hope this helps and that I've read you properly,
Peace
P-Book 1.5, 17" 2gsRAM Mac OS X (10.4.4) FCStudio -
Why can't I update my iPhone 4 to iOS 5.1? Every time I try it gets all the way done downloading it; however when it's processing the file the page tells me "There is a problem downloading the software for the iPhone. Network connection timed out. Make sure your network settings are correct and your network connection is active. or try again later." and next to the file it has err = -3259. I have tried this 17 times and it won't work. My iTunes is completely up to date. I have made sure that my connection is fine and I have tried turning my firewall off with no avail. This has happened to me before but usually the update will go through after 2-3 tries, never to this extent. I don't know what to do. Help!
Error -3259 is a network timeout error, usually. This article might help:
http://support.apple.com/kb/TS2799 -
Hi - I'm a vid beginner, actually have a training session tomorrow at an App Store, but for now...what's the easiest way to cut out about 20 seconds (in the middle) of a minute plus vid that i made on and downloaded from my iPhone 6? I I'mported the clip to iPhoto on my new Macbook pro. The TRIM function in iPhoto seems only to want to let me cut from either end. I guess if I do cut out a section in the middle, maybe 'glueing' the sections back together is too much for iPhoto to handle (?). I spent about 10 minutes just trying to (unsuccessfully) import to iMovie, so that aint working either. Quicktime maybe? Thx for reading
Hey jabXhook,
Thanks for the question. You may have already received your answer from the Apple Store yesterday, but iMovie is definitely the way to go if you want to trim or split clips. Check out the following resource:
Trim, split, and move clips - iMovie Help
http://help.apple.com/imovie/mac/10.0.6/#/movf8b8fc9b2
Trim unwanted frames from a clip
1. In the timeline, select the range of frames you want to keep in the clip.
2. Control-click the clip, and choose Trim Selection from the shortcut menu.
The clip is trimmed to the boundaries of the selection.
Thanks,
Matt M. -
is there any way else to do it without useing quicktime pro....hope apple fixes this in ios 7 the video should come out in landscape regardless of the way you hold your phone or if you start in portrait and go into landscape while recording
You used the data. Verizon can not see what it was sued for. However your phone can see whats apps used the data. go to settings-data usage- there will be a place that says data usage cycle. line the dates up with your cycle. then there will be a bar graph below that extend bother white bars one all the way to the left and one all the way to the right. after those are extended below that will be a list of apps, there should be one that used over 2 gb and that will show you what app used that data in her purse
-
Any way to find out which apps I use the most?
Sometimes there are apps that I don't realize I use a lot. Is there any way to find out which apps I use the most, so I can put those on my first home screen?
I'd keep and eye on your multitask bar. This tells me what apps I use and often forget I've been in that day.
-
How do I tell the user she is out of memory?
In my Swing app, a thread can try and get too much data from the database. It catches the OutOfMemoryError exception, and passes it back up to the GUI. And then.... the GUI crashes when it tried to display the error message, because it's out of memory.
I tried putting System.gc() calls all over the place. No joy.
Ideas?
--- EricRight now, what I have my app doing is warning the user when they initiate an unlimited query (queries have a default limit). The warning basically states that "If too many results are returned, then the app will abruptly exit, because it runs out of memory, and I can't figure out a way to trap this case. Have a nice day, I'm a lousy programmer."
I wonder if there is some way to detect when the vm is running low on memory, but isn't quite out. Then it could abort the process cleanly, tidy up, and keep the app running. Here's the code that runs out of memory:
DataInputStream in = new DataInputStream(connection);
ObjectInputStream oin = new ObjectInputStream(in);
thingy = oin.readObject();Has anyone attempted to subclass ObjectInputStream and override readObject() with readObject(int iAbortThresholdInKilobytes)? Would this be tough? -
How can I solve out of memory error on excell file in PL/SQL
Hi,
I'm new on PL/SQL. One of the PL/SQL code which is created excell report got out of Memory error. The first reason of this error, excell file not supported more than 65536 data. So I change the excell file separeted sheets. So that the single sheet size cannot exceed 65536 data.
All the data are held on system cach and if many user want to take the report the they would get an out of memory error.
So I want to change the code like that; when out of memory exception raises,
the old excell file save to disk and new excell file is created,
and go on to write the new file without exiting the program.
At the end of the data all the excell file append and show only one file to the user.
I do know how to save the file and create a new file. But I don't know how can PL/SQL program to turn back to loop again when the exception occurs.
Is anyone help me on this issue?
Here is my code
Thank you
dworkbook:=hssfworkbook.new;
dCurrentItem := Get_Block_Property(pCurrentBlock, FIRST_ITEM);
while not (name_in('system.last_record')='TRUE') loop
/* The data would be written to the excell file column order. */
if (dRow=0) then
/* Create a new sheet */
elsif (dRow <= dMaxWorksheetNum) then
/* Data of the report are written here. The data are written in column order */
if (dRow > dMaxWorksheetNum) then
/* give dRow and dColumn intial value */
/* increase worksheet number */
end if; /* End of if (dRow=1) */
if (isWritten) and not name_in('system.last_record')='TRUE'then
/* if not at the end of the record and the previously read record is written to the file
, then go to next record */
next_record;
end if;
/* save excell report */
workbookwriter.save(dworkbook,global.gethome||dFileName);
web.show_document('/users/'||dFileName,'_BLANK');
/* when exceptions occurs */
EXCEPTION
WHEN ORA_JAVA.EXCEPTION_THROWN THEN
begin
javaException := ORA_JAVA.LAST_EXCEPTION;
-- Print out the Exception by using the toString()
-- Method of the exception Object
javaException2 := Exception_.new(javaException);
mess(27002,Exception_.getMessage(javaException2));
-- and clean up
ORA_JAVA.CLEAR_EXCEPTION;
exception
WHEN ORA_JAVA.JAVA_ERROR THEN
-- In this case, the error would be
-- Argument 1 can not be null
mess(27002,ORA_JAVA.LAST_ERROR);
--Clean up
ORA_JAVA.CLEAR_ERROR;
end;
WHEN ORA_JAVA.JAVA_ERROR THEN
-- In this case, the error would be
-- Argument 1 can not be null
message(ORA_JAVA.LAST_ERROR);
--Clean up
ORA_JAVA.CLEAR_ERROR;No need to double-post... most questions are answered pretty quickly...
-
CVI 2013 Error: The compiler has run out of memory.
Hello,
I get this error in a source file I'd like to debug:
1, 1 Error: The compiler has run out of memory.
1, 1 Note: You may be able to work around the problem:
1, 1 A. Set the debugging level to 'no run-time checking'.
1, 1 B. Split your source file into smaller files.
1, 1 C. Enable the 'O' option for your source file in the project.
1, 1 D. Move large static data structures into new files and
1, 1 enable the 'O' option for the new files.
Options A and C disable debugging aids mostly, and I don't dare editing it.
So any possibility to increase the memory limit?
/* Nothing past this point should fail if the code is working as intended */
Solved!
Go to Solution.This is the "strange code"
#pragma pack (push,8)
typedef struct
int struct1int;
} STRUCT1;
typedef struct
STRUCT1* s1_ptr;
} STRUCT2;
typedef struct
char c[2];
int i;
STRUCT2 s2;
} STRUCT3;
#pragma pack (pop)
static STRUCT3 s3_global;
void SomeFunc(void)
s3_global.i = 0;
I believe clang fails to compute the struct paddings.
/* Nothing past this point should fail if the code is working as intended */ -
Upgrading to iso 5 and received an error message "the network connection timed out" and fails to complete the upgrade. How do I complete the upgrade?
Try a manual install, as outlined in the link below.
Basic troubleshooting steps
17" 2.2GHz i7 Quad-Core MacBook Pro 8G RAM 750G HD + OCZ Vertex 3 SSD Boot HD
Got problems with your Apple iDevice-like iPhone, iPad or iPod touch? Try Troubleshooting 101
In Memory of Steve Jobs -
I've taken really good care of it and had it since the third gen came out (bought it right away) the screen is cracked from the bottom right corner and im wondering if i can replace it for free or even at all please help
You can replace it, but not for free.
iPad mini: $219
iPad 2 or earlier, iPad mini Retina display: $249
iPad 3rd generation or later: $299
Maybe you are looking for
-
Ipod 'charging' issues & ipod not recognised by itunes/laptop
Hi My ipod was working fine until last week when it appeared to have no battery. Now whenever it is being charged it is always in a 'charging' status despite the fact that it can have been charging for hours and should be fully charged. Also when I u
-
How to prevent from creating multiple deliveries per sales order
Hi, Appreciate your help on the below., I do not want the users to create multiple deliveries for a single sales order. They should only be allowed to create one delivery and it can either be a full delivery or a partial delivery. Is there any possib
-
"Clicking" noise from CD Drive on 12"Powerbook
My 12" Powerbook makes a horrible "clicking" noise when a insert a CD or DVD. After a few seconds the CD or DVD is ejected. This is pretty sudden thing as it was fine one day and now this. Can anyone tell me what may be causing this, or how to fix it
-
Weird problem with vertical text in a table
Hello, I am new on this forum, usually I don't post because I always find the answer to my question, but this is something quite weird... I have a table with vertical text in it (it has to be vertical because the cells are narrow). And I want it to b
-
When I bought a used titanium with tiger installed, I noticed that it did not have iphoto on it. Is there a reason for that and can I find a way to put it on. Without iphoto is there a way to load pictures? Thanks for any help, Jim