Power BI for huge data

Hi,
We are using Power BI for Natural Language Query and reporting. We have to build it for data of around 200 million records. Which is fairly larger in size than 250MB (specified as data size limit for Power BI ). Is there any way to work with this size
of data in Power BI ? Just for reference when we tried it for 54 Millions records and size of data in excel file was 1.8 GB and it did not work. Could anyone please tell us how can Power BI work with this much data.
Thanks and regards,
Arvind
arvind chamoli

Hi Arvind,
Currently Power BI does have a limitation of 250MB, therefore you'll need to think about ways to shrink the data to be stored in the workbook. One practical way is to think about normalization. For example, last week I got a huge XML file from stackoverflow
of 26GB, with a schema like:
date, post content, list of tags, other columns that I am not interested in
I wrote simple code to break it down into three csv files:
date id, date
date id, tag id
tag id, tag name
Basically I got rid of the data that I am not interested e.g. post content, comments, etc. and normalized the data to replace dup strings with id. Hope that helps :-)
Samuel

Similar Messages

Partition merge for huge data

The application has a very large table with more than 5 billion data , using that table it is planned to have only some years data by doing this it is expected to reduce 20 to 25 % data reduced.
The problem here is as it is very large table we plan to get the required data in partition and plan to have this partition merged in to single partition then exchange partition to one table which is not having partition.
My question is how efficiently this partition merge is with huge data volume ?

Hello,
I wonder why are you using Bulk Collect, when the same can be accomplished by a simple Merge statement.
Below can help:
MERGE INTO hotels tgt USING
(SELECT hotel_code,
hotel_name,
hotel_type,
hotel_address,
hotel_number,
hotel_facility
FROM bck_hotel
) src ON (src.hotel_code = tgt.hotel_code)
WHEN MATCHED THEN
UPDATE
SET tgt.hotel_name   = src.hotel_name,
    tgt.hotel_type     = src.hotel_type,
    tgt.hotel_address = src.hotel_address,
    tgt.hotel_number   = src.hotel_number,
    tgt.hotel_facility = src.hotel_facility WHEN NOT MATCHED THEN
INSERT
      tgt.hotel_code,
      tgt.hotel_name,
      tgt.hotel_type,
      tgt.hotel_address,
      tgt.hotel_number,
      tgt.hotel_facility
    VALUES
      src.hotel_code,
      src.hotel_name,
      src.hotel_type,
      src.hotel_address,
      src.hotel_number,
      src.hotel_facility
    );Is it not true?
Regards,
P.

Is Multimaster Timesten replication a good option for huge data volumes?

Hi,
There are 3 timesten nodes in our production setup .There will be around 5 million rows in each node initially which will gradually increase to about 10 million.Once our application moves to production, there will be around 50-70 transactions per second in each node , which need to be replicated on to the other node.
Initially we thought of going with Active-Standby-Subscriber replication.However in this case if active and standby node go down,then it will be a site failure case.So is Active-Active (Multimaster replication) configuration a good option ? Will data collision happen when replication happens in both directions?
Thanks in advance.
Nithya

Multi-master replication is rarely a good idea. You will get data collisions unless you rigorously partition the workload. Conflict detection and resolution is not adequate to guarantee consistency over time. Recovery back to a consistent state after a failure is complex and error prone. I'd strongly advise against a multi-master setup, especially for a high volume system.
You seem to be concerned that 2 out of the 3 systems may fail resulting in a site outage. The likelihood of that is small if you have set things up with separate power etc. With the A/S pair based approach you would still have query capability if the two master systems failed. The chances of all 3 systems failing is not that much less than of just 2 failing in reality I would say (depending on the reason for the failure).
Chris

Internal table for huge data

HI,
I have to populate internal table with huge number of data.
What type of internal is suitable for this?
Regards,
Ram

Hi ram,
As long as you do not have any complex read functionalities, or nested loops, it should be fine to use the normal internal table.
Regards,
Ravi

Best Practise for loading data into BW CSV vs XML ?

Hi Everyone,
I would like to get some of your thoughts on what file format would be best or most efficient to push data into BW. CSV or XML ?
Also what are the advantages / Disadvantages?
Appreciate your thoughts.

XML is used only for small data fields - more like it is easier to do it by XML rather than build an application for the same - provided the usage is less.
Flat files are used for HUGE data loads ( non SAP ) and definitely the choice of data formats would be flat files.
Also XML files are transformed into a flat file type format with each tag referring to the field and the size of the XML file grows to a high value depending on the number of fields.
Arun

Power View in SharePoint Server - The data extension type for a data source is not valid

Hi All,
All of a sudden I am getting following error when trying to create Power View report using shared report data source (no error testing the connection):
"The current action cannot be completed. The data extension type for a data source
'http://dev/Shared Ducuments/Sales.rsds' is not valid for this operation"
I already have a data source (I had created it after creating my site collection a week ago) and when I use this source to create Power View report then there is no error but I am getting above error when I create another similar data source and use it to create
a Power View report.
Please help me to resolve the error.
Thanks

I am going nuts! I had selected 'Analysis Services' instead of 'Microsoft BI Semantic Model for Power View'

HT3180 Hi, I've tried to re start mt apple TV after relocating. I get the Apple logo on power up & then it searches for time & date. After a few minutes of doing this, the screen goes blank. I have no menu screen - nothing??

I have no idea what to do? I can't reset my Apple TV because I cant access settings. Mr remote is blinking three times indicating it's not paired, so I can't reset it that way!
1. Power on and Apple Logo aphears.
2. TV trys to find time and date ( not the new start up language, location etc)
3. Screen on TV is blank
4. Try remote and Apple TV just blinks three times.
WIFI Network is pefect & pairs with all my IT
Please help

Obviously, your network isn't perfect.
Assuming this is not the first time you have used your Apple TV
You might try restarting the Apple TV by removing ALL the cables for 30 seconds.
Also try restarting the router. (Also try removing it’s power cord for at least 30 seconds)
If the problem persists, try a restore, you may want to try the previous procedures several times before doing this.
If restoring from the Apple TV doesn't help, try restoring from iTunes using a USB cable.
If this is a new Apple TV, in addition to trying the above, it may also be that your network router is not allowing access to the timeserver, check that your router allows access over port 123.

Method for Downloading Huge Data from SAP system

Hi All,
we need to download the huge data from one SAP system & then, need to migrate into another SAP system.
is there any better method, except downloading data through SE11/SE16 ? please advice.
Thanks
pabi

I have already done several system mergers, and we usually do not have the need to download data.
SAP can talk to SAP. with RFC and by using ALE/IDOC communication.
so it is possible to send e.g. material master with BD10 per IDOC from system A to system B.
you can define that the IDOC is collected, which means it is saved to a file on the application server of the receiving system.
you then can use LSMW and read this file with several hundred thousand of IDOCs as source.
Field mapping is easy if you use IDOC method for import, because then you have a 1:1 field mapping.
So you need only to focus on the few fields where the values changes from old to new system..

Data source for this data connection isn't registered for Power BI

Hi, I am getting this error when I set Schedule Data Refresh to refresh data from db. How do I register my connection to the data source? Is this a fix on SQL server or Power BI?
FAILURE INFORMATION
Failure
Correlation ID: c5132b7a-3c54-4f12-a048-3ddcf0b95f26
Data connection Status
SqlServer twcperfsql1 OfficePerf OnPremise error: Sorry, the data source for this data connection isn't registered for Power BI. Tell your Power BI admin the admin needs to register this connection to this on-premises data source.
Thanks for your help.

I'm running into the same issue, I did configure the Gateway and then the data source in Power Bi.
When I go to the Power BI Admin Center and click on "Test Connection" it seems to be working. But when I refresh the data from Power BI it doesn't work. Now if I edit the spreadsheet in excel and refresh the data in PowerPivot then it works. So
not sure why the solution doesn't refresh from the automated service in Power BI without this useless error.
Thanks
Fabian
In case it helps this is the actual error:
Failure Correlation ID: ecc73a16-7264-45b2-9baf-e3448f007211
Power Query - dbo_TableAOnPremise error: Sorry, the data source for this data connection isn't registered for Power BI. Ask your Power BI admin to register the data source in the Power BI admin center.
Also further information I have found so far is that the Gateway is receiving the requests for the refresh and it shows the message below, so the data source is found and communication seems to be happening from server to client and back, not sure why server
doesn't seem to like the data the Gateway is sending back to server.
The feed 'dbo_TableA' was successfully accessed at 7/15/2014 4:23:26 PM.
The request took 00:00:00.0000105 seconds.
Activity ID: e8464e5d-3f0a-49c2-b10b-450fec3e5940

Table maintenence Generator for a Huge data Table.

Hi Experts,
I have created a Table maitenance for a Custom table which has some 80,000,000 records approx.
Now the when i run this, it goes for a short dump saying "STORAGE_PARAMETERS_WRONG_SET".
The basis have reported that the tcode is running a sequential read on the table & they are saying that is the reason for this dump.
Are there any limitations that Table maintenance can't be created for table with huge data?
Or should the program be modified to accomodate the "READ" from the tables in case of large entries?
Please advice.
regards,
Kevin.

Hi,
I think this is because of memory overflow.
You can create two screens for this, in on screen (Overview) screen, restrict the data selection.
In detail screen display the data.
With regards,
Vamsi

Migration of huge data from norm tables to denorm tables for performance

We are planning to move the NORM tables to DENORM tables in Oracle DB for a client for performance issue. Any Idea on the design/approach we can use to migrate this HUGE data (2 billion records/ 5TB of data) in a window of 5 to 10 hrs (Or minimum than that also).
We have developed SQL that is one single query which contains multiple instance of same table and lots of join. Will that be helpful.

Jonathan Lewis wrote:
Lother wrote:
We are planning to move the NORM tables to DENORM tables in Oracle DB for a client for performance issue. Any Idea on the design/approach we can use to migrate this HUGE data (2 billion records/ 5TB of data) in a window of 5 to 10 hrs (Or minimum than that also).
We have developed SQL that is one single query which contains multiple instance of same table and lots of join. Will that be helpful.Unfortunately, the fact that you have to ask these questions of the forum tells us that you don't have the skill to determine whether or not the exercise is needed at all. How have you proved that denormalisation is necessary (or even sufficient) to solve the client's performance problems if you have no idea about how to develop a mechanism to restructure the data efficiently ?
Regards
Jonathan LewisYour brutal honesty is certainly correct. Another thing that is concerning to me is that it's possible that he's planning on denormalizing tables that are normalized for a reason. What good is a system that responds like a data warehouse but has questionable data integrity? I didn't even know where to begin with asking that question though.

Delta data load for huge table in Data Services XI R3

Hi,
We have a project requirement of pulling delta data once in a week from table which has around 44 million records but it doesn't have any last modified column.
In such case we have to use table comparison transform, but it will be very time consuming.
Please provide some suggestions to meet this requirement.
The source of DS job is a snapshot(the source table is in a remote database).
Thanks!

Because SAP Business Objects Data Services XI 3.x doesn't have any built-in delta-enablement mechanism with R3/ECC, the only possibility right now is to do a table compare. Given that this is only a weekly activity, while maybe time-consuming, it may be sufficient.
In the future, you may want to consider activating the related SAP BW standard content DataSource(s) to extract into Data Services prior to loading to the target repositories. This may provide delta-enablement for the data that you're extracting.

POWER QUERY Get External Data From File From Folder (Excel 2013)

Hi,
Beginner's question :
What could be the use of the query on a folder : we just get a list of files with their path. What can we do with that?
Thanks

Hi,
Do you want to combine data from multiple Excel Files in the same folder path into one table? If I understand correct, we can add a custom column to import the data.
After we getting a list of files with their path, the Query Editor window will activate to show you a table containing a record for each file in the chosen directory. These will provide our function with the needed FilePath and FileName parameters.
Function sample: File name([Folder path],[Field name]
For more detailed steps, please see the article:
http://datapigtechnologies.com/blog/index.php/using-power-query-to-combine-data-from-multiple-excel-files-into-one-table/
Please Note: Since the web site is not hosted by Microsoft, the link may change without notice. Microsoft does not guarantee the accuracy of this information.
George Zhao
TechNet Community Support
It's recommended to download and install
Configuration Analyzer Tool (OffCAT), which is developed by Microsoft Support teams. Once the tool is installed, you can run it at any time to scan for hundreds of known issues in Office
programs.

Maintaining Remote key in look up tables for Production data

Hi,
me
SEE in DEV and QA to maintain the Remote key for Look up tables.First i have exported those look up table values and again imported with mapping Remote keys (clone the field and map it to Remote key.). By this i have maintained the Remote keys to all look up tables. Now for Production what should i do. Is it the same process extract all records from production into our desktop and then again import it using IM, clone the field and map it to Remote key.
1. Is this the right process?
2. What happens if there are huge data present in production how to proceed furthr then?
Required your suggestion please.
Thanks,
Pradeep

HI Pradeep,
I dont see any problem with this method if I am getting it right
You are updating remote keys by reimporting lookup data, and this way you have multiple remote keys for data in lookup table,right?
This will help you to automap entries in Import process for different Remote systems.
I did not get your 2nd point?
Thanks,
Ravi

Powerpivot for sharepoint error: Unable to refresh data for a data connection in the workbook

Hello,
I have three errors when i try to use a simple powerpivot workbook published in sharepoint: (nothing on google has help me..)
1-Unable to refresh data for a data connection in the workbook.
Try again or contact your system administrator. The following connections failed to refresh:
PowerPivot Data
2-The embedded PowerPivot data in the workbook cannot be loaded due to a version mismatch
3-01/21/2012 17:26:47.08 w3wp.exe (0x1950)                       0x0AD0 Excel Services Application    Excel
Calculation Services    bccc Medium Session.HandleTrimmedWorkbookReloading: userOperation ApplySlicerSelectionOperation requires BaseWorkbook: "http://crm2011:2020/Marketing%20Reports/test2_excel32bits.xlsx"
[0x409] [Saturday, 21 January 2012 09:40:18] [BaseWB ID: 2] to be untrimmed if it is currently trimmed. The workbook is currently NOT trimmed. fb614a65-e398-4b97-a98d-fb7b23eab39f
01/21/2012 17:26:47.08 w3wp.exe (0x1950)                       0x0AD0 Excel Services Application    Excel Calculation
Services    f1va Medium CWorkbookWrapper::CWorkbookWrapper: Created with ID=4 fb614a65-e398-4b97-a98d-fb7b23eab39f
01/21/2012 17:26:47.09 w3wp.exe (0x1950)                       0x0AD0 Excel Services Application    Excel Calculation
Services    eq3r Medium ConnectionRequest.ConnectionRequest: New connection request. SessionId=1.V21.4PI+fCwIq52LH++nOoMzs90.5.en-US5.en-US73.-0060#0000-10-00-05T03:00:00:0000#+0000#0000-03-00-05T02:00:00:0000#-006036.bfceb31b-7122-46ca-9e2a-ae52cefcfcaf1.N,
WorkbookVersion=ConnectionInfo.WorkbookVersion: Uri=http://crm2011:2020/Marketing Reports/test2_excel32bits.xlsx, Version=Saturday, 21 January 2012 09:40:18 fb614a65-e398-4b97-a98d-fb7b23eab39f
01/21/2012 17:26:47.12 w3wp.exe (0x1950)                       0x0AD0 Excel Services Application    Excel Calculation
Services    aysl Medium Succeeded to initialize a chart. fb614a65-e398-4b97-a98d-fb7b23eab39f
01/21/2012 17:26:47.12 w3wp.exe (0x1950)                       0x0AD0 Excel Services Application    Excel Calculation
Services    8xk9 Medium ExternalSource.ExecuteOperation: We exhausted all available connection information. Exception: Microsoft.Office.Excel.Server.CalculationServer.Interop.ConnectionInfoException: Exception of type
'Microsoft.Office.Excel.Server.CalculationServer.Interop.ConnectionInfoException' was thrown.     at Microsoft.Office.Excel.Server.CalculationServer.ConnectionInfoManager.GetConnectionInfo(Request request, String externalSourceName, Int32
externalSourceIndex, Boolean& shouldReportFailure)     at Microsoft.Office.Excel.Server.CalculationServer.ExternalSource.ExecuteOperation(Request request, ExternalSourceStateInfo externalSourceStateInfo, ExternalSourceStateInfo prevExternalSourceStateInfo,
Int32 index, ConnectionInfoManager connectionInfoManager, ExternalDataScenario scenario, DataOperation dataOpe... fb614a65-e398-4b97-a98d-fb7b23eab39f
01/21/2012 17:26:47.12* w3wp.exe (0x1950)                       0x0AD0 Excel Services Application    Excel Calculation
Services    8xk9 Medium ...ration, Boolean verifyPreOperationConnection), Data Connection Name: PowerPivot Data, SessionId: 1.V21.4PI+fCwIq52LH++nOoMzs90.5.en-US5.en-US73.-0060#0000-10-00-05T03:00:00:0000#+0000#0000-03-00-05T02:00:00:0000#-006036.bfceb31b-7122-46ca-9e2a-ae52cefcfcaf1.N,
UserId: 0#.w|contoso\manager fb614a65-e398-4b97-a98d-fb7b23eab39f
My server and client olap versions are the same: MSOLAP.5, i used sql server 2008 R2 SP1 and sharepoint 2010 SP1 and reboot or iisreset have no effect
Thanks in advance for your help

Hello Challen Fu
I would be so grateful if you could please help me out
I have been trying to find a solution to the same error message
In my case, the power pivot reports were working before on a regular team site , but then two things changed:
a) I created a toplevel site using the BI Center template. Now I am using a Business Intelligence template , created a power pivot gallery library and uploaded a few powerpivot reports
b) On the  backend, the database instance was upgrated to SQL Server 2012
     Front end Server VDSP01 remains SQL Server 2008 R 2 where Sharepoint 2010 was installed as a FARM
Now, the reports will display in sharepoing however they will not refresh. the error message i get is the same.
Scenario recap:
a- Server VDSP01 uses SQL Server 2008 R 2 where Sharepoint 2010 was installed as a FARM
b- On the back end, the database instance name was replaced with SQL 2012 Server:
         from SQL Server 2008 R 2 (instance DBDEV-COTS\COTS)
           to     SQL Server 2012 ( instance VTSQL01\COTS)
c-  I was told that:
     From VDSP01, they ran
CliConfg.exe to create SQL Server Alias :
       where    BEFORE: vdsharepoint --> DBDEV-COTS\COTS
            and AFTER    : vdsharepoint --> VTSQL01\COTS
I appreciate in advance any help you can provide<v:shapetype coordsize="21600,21600" filled="f" id="_x0000_t75" o:preferrelative="t" o:spt="75" path="m@4@5l@4@11@9@11@9@5xe" stroked="f">
<v:stroke joinstyle="miter">
  <v:formulas>  <v:f eqn="if lineDrawn pixelLineWidth 0">
  <v:f eqn="sum @0 1 0">
  <v:f eqn="sum 0 0 @1">
  <v:f eqn="prod @2 1 2">
  <v:f eqn="prod @3 21600 pixelWidth">
  <v:f eqn="prod @3 21600 pixelHeight">
  <v:f eqn="sum @0 0 1">
  <v:f eqn="prod @6 1 2">
  <v:f eqn="prod @7 21600 pixelWidth">
  <v:f eqn="sum @8 21600 0">
  <v:f eqn="prod @7 21600 pixelHeight">
<v:f eqn="sum @10 21600 0">
</v:f></v:f></v:f></v:f></v:f></v:f></v:f></v:f></v:f></v:f></v:f></v:f></v:formulas>
<v:path gradientshapeok="t" o:connecttype="rect" o:extrusionok="f">
<o:lock aspectratio="t" v:ext="edit">
</o:lock></v:path></v:stroke></v:shapetype> <v:shape alt="" id="Picture_x0020_2" o:spid="_x0000_i1025" style="width:630pt;height:475.5pt;" type="#_x0000_t75">
<v:imagedata o:href="cid:[email protected]" src="file:///C:\Users\wlarange\AppData\Local\Temp\msohtmlclip1\01\clip_image001.jpg">
</v:imagedata></v:shape>
wanda larangeira

Power BI for huge data

Similar Messages

Maybe you are looking for