'Merge' function in Power Query

A bit of background-
I have 3 .txt files that I would like to combine into one table in Excel using Power Query. Two of the files contain the same kind of data, and the third contains geographic data I'd like to add on. The first two files I mentioned have an 'ID number' column
that is also contained in the third file (with geographic data). My goal is to append the first two files, and then use a merge to add geographic fields to the data, matching up based on common ID numbers.
The first file is about 10000 rows, the second is about 6000. The files are not very big, however when I try to expand the columns in the merge, it says that Excel has loaded several Gbs from the file I'm trying to merge with, when the file is only about
16 mb.
Anyone have any ideas how this merge could cause all this 'data' to be loaded, never allowing it to be completed?

Hi,
Thanks for your feedback. One way that you could approach this scenario is by importing the three files using "From File -> From Text" in Power Query, then use "Append" to get a new query that appends the first two queries, and finally do a Merge
operation between the result of Append and the third file.
If you are indeed doing this and still hitting the issue, you can send us more details via "send a frown" and selecting "include formulas", which will give us more information to help you debug the issue.
Thanks,
M.

Similar Messages

Add actions to an existing query or function in Power Query

Hello
I have just started using Power Query and can see this will be an amazing tool to add to my tool set. However I am rather slow on the uptake and am having trouble with some of the basics of using Power Query
I have successful created some queries and by following along on some articles at excelguru.ca/blog i have created a usable function or two
The issue i have and can see this being something i will do a lot in the near future (especially while still learning), is that I need to go back to the query and function and edit it. In particular i need to either rename some columns to make them compatible
with other data sources, or delete them.
I can see that i can manually edit the query / function in the advanced editor, but i was wondering if it is possible to open the report up again and edit the columns in the same manner as when creating them in the first place?
Thanks in advance for any help given.
I am using Excel 2010, although do have access to Excel 2013 if needed, but as i am using PowerPivot in 2010 i would prefer to use PQ in 2010 also.
Proportal

OK - What had completely stumped me, and i had to set it aside to do other things, has just been solved!
To explain - obviously editing Queries is simple - right click the query and edit... and then make the changes you want.
For functions - the thing i was really struggling with is also fairly simple, but takes a few more steps.
1 - copy the query from the advanced editor
2 - create a new blank query from Get External Data on the Home Tab / From Other Sources / Blank Query
3 - Past your function in and remove the function statement(s) and reinstate and file path information
4 - Edit the columns etc as you want.
5 - when completed paste the relevent sections back into the original function.
Done.
Hope this helps any others out there that were struggling with something so simple...
Moral of this - copy and save all your queries and functions in a text editor such as notepad++ so you can cut out a lot of the steps above...

Dcount function in power query

Hi,
I have a query generated in power query
that gives me the following results:
ID          VALUES
1              A001
2              A002
3              A001
4              B001
5              B002
6              B001
I need to insert a new calculated field
in the query I of the order of
repetition of each record. The result we
want to achieve is the following:
ID          VALUES       RESULT
1              A001             1
2              A002             1
3              A001             2
4              B001             1
5              B002             1
6              B001             2
Using access get it using the following
calculated field in a query: " DCont("*";"tabla1";"Valor='" & [valor] & "'and Id_prueba<=" & [id_prueba])".
How I can do the same with
power query?
Thank you very much in advance and Merry Christmas
to all,
José Luis

Hi,
You can can use Table.Group and Table.AddIndexColumn function to solve this.
let
Source = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
Group = Table.Group(Source, {"VALUES"}, {{"Group", each Table.AddIndexColumn( Table.Sort(_, {"ID", 0 } ) , "RESULT" ,1,1 ), type table}}),
Final = Table.Sort( Table.Combine( Group[Group] ) , { "ID", 0 } )
in
Final
Regards,
Hi,
First thank you very much
for your answer. Could you please tell me
where I have to enter the code
you indicate in your response ?, the
query´s code where I need to create the
RESULT field is as follows:
let
    Origen = Xml.Tables(Web.Contents("http://localhost:xxxx/planning_units/1001/schedule/2014-09-29.xml?end_date=2015-12-31&level=schedule&auth=xxxxxxxxxxxxxxxx=")),
    #"Tipo cambiado" = Table.TransformColumnTypes(Origen,{{"Attribute:type", type text}}),
    #"Expandir schedule" = Table.ExpandTableColumn(#"Tipo cambiado", "schedule", {"activity-id", "booking-date", "employee-id"}, {"schedule.activity-id",
"schedule.booking-date", "schedule.employee-id"}),
    #"Expandir schedule.activity-id" = Table.ExpandTableColumn(#"Expandir schedule", "schedule.activity-id", {"Element:Text"}, {"schedule.activity-id.Element:Text"}),
    #"Expandir schedule.booking-date" = Table.ExpandTableColumn(#"Expandir schedule.activity-id", "schedule.booking-date", {"Element:Text"}, {"schedule.booking-date.Element:Text"}),
    #"Expandir schedule.employee-id" = Table.ExpandTableColumn(#"Expandir schedule.booking-date", "schedule.employee-id", {"Element:Text"}, {"schedule.employee-id.Element:Text"}),
    #"Índice agregado" = Table.AddIndexColumn(#"Expandir schedule.employee-id", "Índice", 1, 1),
    #"Columnas reordenadas" = Table.ReorderColumns(#"Índice agregado",{"Índice", "schedule.activity-id.Element:Text", "schedule.booking-date.Element:Text",
"schedule.employee-id.Element:Text", "Attribute:type"}),
    #"Columnas con nombre cambiado" = Table.RenameColumns(#"Columnas reordenadas",{{"Índice", "ID"}}),
    #"Columnas quitadas" = Table.RemoveColumns(#"Columnas con nombre cambiado",{"Attribute:type"}),
    #"Columnas reordenadas1" = Table.ReorderColumns(#"Columnas quitadas",{"ID", "schedule.booking-date.Element:Text", "schedule.activity-id.Element:Text",
"schedule.employee-id.Element:Text"}),
    #"Personalizada agregada" = Table.AddColumn(#"Columnas reordenadas1", "VALUE", each [#"schedule.booking-date.Element:Text"]&[#"schedule.activity-id.Element:Text"]&[#"schedule.employee-id.Element:Text"])
in
    #"Personalizada agregada
Thank you very much in advance,
José Luis

Pivot Function in Power Query

Hi all,
One thing i'm struggling with is the use of the pivot function to pivot a column within PQ. I'm confused on why this happens and why I can't clear it up...but here goes:
I unpivot a set of columns which then gives me two columns (Attribute/Value). I do this because I have repeated columns of data, but it should all fall under one column (think Hours, Hours1, Hours2, etc) - basically the input is coming from a form
that has repeating fields but everything is the same data type.
I make some adjustments to the Attribute columns (Replace items within the Attribute column) then Pivot the two columns again...theoretically I would be able to do this since i just unpivoted the same data, but:
I get tons of errors (usually "There were too many elements in the enumeration to complete the operation".). I'm not entirely sure why this happens and it doesn't seem to make sense to me that I can't UnPivot and Pivot and get the same results
as I originally had.
Any recommendations would be greatly appreciated!
EDIT: I'm choosing "don't aggregate" during the Pivot operation...I get values if I keep some sort of aggregate function, but not the right values (I get a count of the fields instead of the data within the fields)

Try this solution:
Once you are to step 3, add an index column that starts at 0 and another index column that starts at 1.
Transform by Pivot Column using the names in Attribute to create new columns and the Values Column set to Value. Under advanced options select Don't Aggregate.
Fill Down the Date column
Filter rows to remove null from Work Time.
Here's the M code from the advanced editor:
let
    Source = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
    #"Added Index" = Table.AddIndexColumn(Source, "Index", 0, 1),
    #"Added Index1" = Table.AddIndexColumn(#"Added Index", "Index.1", 1, 1),
    #"Pivoted Column" = Table.Pivot(#"Added Index1", List.Distinct(#"Added Index1"[Attribute]), "Attribute", "Value"),
    #"Filled Down" = Table.FillDown(#"Pivoted Column",{"Date"}),
    #"Filtered Rows" = Table.SelectRows(#"Filled Down", each ([Work Time] <> null)),
    #"Changed Type" = Table.TransformColumnTypes(#"Filtered Rows",{{"Work Time", type time}})
in
    #"Changed Type"
If that doesn't work, this example might do the trick:
http://www.excelguru.ca/blog/2015/01/07/transpose-stacked-tables/

Power Query filters based on UserID

I have a sales dashboard built with Power Query, Power Pivot, Power View and Power Maps that I need to filter based on the user that opens the dashboard. Users in ABCD territory, should see sales data for ABCD sales, but should not see sales from ZYXW territory.
We do this all the time in SSRS using User!UserID as a parameter passed into a stored procedure, but I am not familiar with what function within Power Query would pass the user ID as a parameter into the stored procedure.
Any help would be greatly appreciated.

I have solved this limitation by automatically producing one file per user on the dashbord. This also reduce the amount of data per user so the response is fast.
In the database MS SQL there are a number of "temporary" tables which are sources to the Power Pivot
database. Those tables are loaded with subsets of data per user and the database is refresh and published
on the dashbord (Power View). The tables are loaded by stored procedures with the users ID as parameter. Programmed with VBA. No SharePoint in this case.
Gosta Munktell

Table.Join/Merge in Power Query takes extremly long time to process for big tables

Hi,
I tried to simply merge/inner join two big tables(one has 300,000+ rows after filtering and the other has 30,000+ rows after filtering) in PQ. However, for this simple join operation, PQ took at least 10 minutes (I killed the Query Editor after 10
minutes' processing) to load the preview.
Here's how I did the join job: I first loaded tables into the workbook, then did the filtering for each table and at last, used the merge function to do the join based on a same field.
Did I do anything wrong here? Or is there any way to improve the load efficiency?
P.S. no custom SQL was used during the process. I was hoping the so called "Query Folding" can help speed the process, but it seems it didn't work here.
Thanks.
Regards,
Qilong

Hi!
You should import the source tables
in Access. This will speed up the work of
PQ in several times.

Automatic Refresh of a Merged Power Query giving 'Table Not Registered' error

Scenario: I have 5 tables in my Azure VM SQL database associated with my ERP system. I have loaded all 5 tables into a 'Build Excel' workbook, merged the tables together and created a dataset that my users will be using. From that 'Build
Excel' workbook - I published to the BI catalog a data set simply called 'Customer information and Statistics' - This is searchable by my users and they don't have to go through the effort of combining the 5 tables.
I have a published workbook that uses the 'Customer Information and Statistics' data set and nothing else. I tried to schedule a refresh of the data and it failed (below is the error when I did a manual refresh):
OnPremise error: Sorry, the data source for this data connection isn't registered for Power BI. Ask your Power BI admin to register the data source in the Power BI admin center. An error occurred while processing table 'Customer Information and Statistics'.
The current operation was cancelled because another operation in the transaction failed.
So, going down that rabbit hole, I began the process of 'registering' this in Power BI - however, when I copied in the connection string it began walking me through a process of providing credentials to the 5 tables in my SQL server - that are already exposed
in my main data Sources. I figured I would continue and tried to use the service account credentials that my main data sources use and it would not properly authenticate.
Taking a step back it prompted a few questions:
Why am I being asked to register these 5 merged data sources that are already exposed in my base data sources?
Is my approach of creating 'Friendly named' Merged datasets for my users an incorrect approach?
Is there a different method I should be using to establish autorefresh on Merged datasets in these published workbooks using Power Query?
Thanks for the help -
Tom Grounds

Tom, can you submit this as a bug in the UI via the Smile Frown button?
Thanks!
Ed Price, Azure & Power BI Customer Program Manager (Blog,
Small Basic,
Wiki Ninjas,
Wiki)
Answer an interesting question?
Create a wiki article about it!

Need to install Power Query with Internet Explorer 8

Hello, could someone please help me?
I need to install Microsoft Power Query to allow me to easily merge excel tables from different worksheets (files).
The problem is that I am restricted to Internet Explorer 8.
Here are my specs, I cannot change these:
Windows 7, 64-bit operating system
Excel 32-bit, Professional Plus 2010
Internet Explorer 8
When I try to install Power Query, I get the message that I need IE 9.
I and others have other browser-based tools that only work on IE 8.
There must be some work-around for this.
I do NOT require the internet-searching feature of Power Query.
Here are some possible options?
Go into Control Panel and manually switch IE versions each time (not a feasible method at all)
Force Power Query to work using Chrome or FireFox instead
Disable certain features of Power Query to allow installation with IE 8
This is really disappointing, I don't understand why we can't just have Power Query install regardless of IE version.
Some functions could be disabled, but at least we could still use the actually important tools.
Thanks for your help. Any input is appreciated.

Hi,
As far as I know,
Microsoft Power Query for Excel requires Internet Explorer 9 or greater. Windows operation system will check the system requirements before starting installation. Thus, we could not do the options which you mentioned above.
Then, we may try to find the workaround to get your original purpose: Merge excel tables from different worksheets (files).
1. Try to use
PowerPivot
http://www.powerpivotpro.com/2012/09/combine-multiple-worksheetsworkbooks-into-a-single-powerpivot-table/
Please Note: Since the web site is not hosted by Microsoft, the link may change without notice. Microsoft does not guarantee the accuracy of this information.
2. Try to use VBA code
http://msdn.microsoft.com/en-us/library/office/gg549168(v=office.14).aspx
3. If you have Excel 2013, we may try the new featuer "Data Model " in Pivot table:https://support.office.com/client/Tutorial-PivotTable-data-analysis-using-a-Data-Model-in-Excel-2013-f9ad8310-3b5b-48a3-913d-5cc1b364ee46
Hope it's helpful.
Regards,
George Zhao
TechNet Community Support
It's recommended to download and install
Configuration Analyzer Tool (OffCAT), which is developed by Microsoft Support teams. Once the tool is installed, you can run it at any time to scan for hundreds of known issues in Office
programs.

Visual studio online - Excel power query usage and samples needed

Background
Visual Studio Online does not offer much in the way of reporting, when compared to (onsite) TFS... unless you utilize the REST API functionality.
The Visual Studio Online REST API is fairly extensive, but not very 'reporting' friendly to applications like Microsoft Excel.
Question
Are there any good examples of Microsoft Excel Power Query usage - to acquire and create reports via the Visual Studio Online REST API? I've searched - but surprisingly haven't found anything of substance. It is fairly simple to call a VSO REST
query using Excel Power Query, but not so simple to iterate through the lists within lists, etc... unless you are familiar with 'M language'.

Hi,
Thank you for reaching out to us. I am currently researching to gather more information with regards to your request. I shall revert back to you with an update at the earliest. Sincerely appreciate your patience.
Regards,
Nithin Rathnakar

POWER QUERY Get External Data From File From Folder (Excel 2013)

Hi,
Beginner's question :
What could be the use of the query on a folder : we just get a list of files with their path. What can we do with that?
Thanks

Hi,
Do you want to combine data from multiple Excel Files in the same folder path into one table? If I understand correct, we can add a custom column to import the data.
After we getting a list of files with their path, the Query Editor window will activate to show you a table containing a record for each file in the chosen directory. These will provide our function with the needed FilePath and FileName parameters.
Function sample: File name([Folder path],[Field name]
For more detailed steps, please see the article:
http://datapigtechnologies.com/blog/index.php/using-power-query-to-combine-data-from-multiple-excel-files-into-one-table/
Please Note: Since the web site is not hosted by Microsoft, the link may change without notice. Microsoft does not guarantee the accuracy of this information.
George Zhao
TechNet Community Support
It's recommended to download and install
Configuration Analyzer Tool (OffCAT), which is developed by Microsoft Support teams. Once the tool is installed, you can run it at any time to scan for hundreds of known issues in Office
programs.

Can I use Power Query to Import a table from Excel sheet range which starts not from the top row?

Hi,
Being an experienced Excel user before Power BI, I am just starting to explore the M and Power Query capabilities, and need help already (ain't easy to google this use case somehow):
I need to import the table which sits in the Excel file with header row in the row 17 of Excel sheet, with some metadata header in the preceding rows of the columns A and B.
01: Report name, Quick Report
02: Report Date, 1/1/2014
17: Employee Name, Manager, etc...
18: John Doe, Matt Beaver, etc.
Both (a) direct attempt to load as Excel file and (b) the indirect way through [From Folder] and formula in custom column -- both lead to the same error: "[DataFormat.Error] External table is not in the expected format."
Specifically, I tried to use the [Power Query -> From File -> From Folder] functionality, select an Excel file and add a custom column to access the binary content: [Add Custom Column] with formula "=Excel.Workbook([Content])".
It looks like Power Query expects a rectangular range with headers full-width followed by a contiguous table range to import anything, and refuses to load if that is not the case...
QUESTION: Is there any way to load whatever-formatted data from Excel first, and then manipulate the overall imported range (like referring to rows starting from 17th using "Table.SelectRows" etc.) to read the actual data? Reading and using
the metadata from header would be a bonus, but that comes second... The main issue is to get something from a non-regular Excel file to later work with using M formulae ...
Thanks!
SAM

Finally found the answer to this one in ():
You Cannot Open a Password-Protected Workbook
If the Excel workbook is protected by a password, you
cannot open it for data access, even by supplying the correct password with
your connection settings, unless the workbook file is already open in the
Microsoft Excel application. If you try, you receive the following error
message:
Could not decrypt file.
ANSWER: So, will have either weave in the work with temporary unprotected files or requires opening them before updating the data source (although this almost defeats the purpose of automation...)
ANSWER to ORIGINAL QUESTION: password was preventing Power Query from reading the Excel file. For solution see above.
Thanks anyway for participation and inspiration, Imke!

I cannot use Online Search in my Power Query

When I'm using the power query, the "search online" is disable. Do you know how to active this function? Thanks!

Some questions:
1) If you are not already using the latest version of Power Query, can you try update to the latest and retest?
2) Can you provide a screenshot? Also any tooltip popup when you move mouse over the disabled text/icon?

POST http(s) request with xml content in Power Query

I have a POST http request which works good in HttpRequester (Firefox plugin). I know that it is possible to call POST request in Power Query but cannot find working solution. Here is the original request:
POST https://svcs.ebay.com/FeedbackService
X-EBAY-SOA-OPERATION-NAME: createDSRSummaryByPeriod
X-EBAY-SOA-SERVICE-VERSION: 1.0.0
X-EBAY-SOA-SERVICE-NAME: FeedbackService
X-EBAY-SOA-APP-NAME: ***
X-EBAY-SOA-SITEID: 0
X-EBAY-SOA-SECURITY-TOKEN: ***
Content-Type: text/xml
<?xml version="1.0" encoding="UTF-8"?>
<createDSRSummaryByPeriodRequest xmlns="http://www.ebay.com/marketplace/services">
<dateRange>
<dateFrom>2013-12-01T00:00:00.000Z</dateFrom>
<dateTo>2013-12-31T23:59:59.999Z</dateTo>
</dateRange>
<dateRangeEventType>ListingDate</dateRangeEventType>
</createDSRSummaryByPeriodRequest>
Power Query WebContent() function is capable to handle headers and xml content, however, there is no any documentation about its syntax.
In Power Query I use this request:
= Xml.Tables(Web.Contents(“https://svcs.ebay.com/FeedbackService”,
[Headers =[" X-EBAY-SOA-OPERATION-NAME"= "createDSRSummaryByPeriod",
"X-EBAY-SOA-SERVICE-VERSION="1.0.0",
"X-EBAY-SOA-SERVICE-NAME"="FeedbackService",
"X-EBAY-SOA-APP-NAME"="***","X-EBAY-SOA-SITEID"="0",
"X-EBAY-SOA-SECURITY-TOKEN"="***"],
Content=["<?xml version="1.0" encoding="UTF-8"?>
<createDSRSummaryByPeriodRequest xmlns="http://www.ebay.com/marketplace/services">
<dateRange>
<dateFrom>2013-12-01T00:00:00.000Z</dateFrom>
<dateTo>2013-12-31T23:59:59.999Z</dateTo>
</dateRange>
<dateRangeEventType>ListingDate</dateRangeEventType>
</createDSRSummaryByPeriodRequest>"]]))
However, it returns "Expression.SyntaxError: Invalid identifier." Maybe anyone has at least examples of analogous working syntax?

Task is solved. First of all I updated PowerQuery since previous version does not support Headers option for Web.Contents function. Here is a correct syntax:
=Xml.Document(Web.Contents("https://api.ebay.com/ws/api.dll",[Headers=[#"X-EBAY-API-COMPATIBILITY-LEVEL"= "871",#"X-EBAY-API-SITEID"="0",#"X-EBAY-API-CALL-NAME"="GetMyeBaySelling",#"CONTENT-TYPE"="text/xml"], Content=Text.ToBinary("<?xml version="&Character.FromNumber(34)&"1.0"&Character.FromNumber(34)&" encoding="&Character.FromNumber(34)&"utf-8"&Character.FromNumber(34)&"?>
<GetMyeBaySellingRequest xmlns="&Character.FromNumber(34)&"urn:ebay:apis:eBLBaseComponents"&Character.FromNumber(34)&">
<RequesterCredentials><eBayAuthToken>***</eBayAuthToken></RequesterCredentials><SoldList><DurationInDays>60</DurationInDays><Pagination><EntriesPerPage>100</EntriesPerPage><PageNumber>1</PageNumber></Pagination></SoldList></GetMyeBaySellingRequest>")]))

Power Query for Google Sheets

Does anyone know of any plans to create an add-on for Power Query in Google Sheets? I have played with the IMPORTHTML function but it doesn't provide the same data manipulation tools the PQ does. I haven't had any luck finding anything in Google
searches.
Thanks

Here's a way you can do this without a built-in connector:
Make your Google Docs workbook visible to anyone that has the link (or public).
Go File > Download as > Microsoft Excel.
Get the link that was used in the browser's download manager.
Go in PQ > From Web.
Paste the link.
Click "OK"
The workbook will be download as an Excel workbook and opened in PQ.

Power Query for Excel - Need Help with Oracle SQL Syntax

Hello everyone,
I am new to Power Query and am not able to figure this out. I am trying to pull in data into my Excel spreadsheet using a specific Oracle SQL query. While in query editor, how do I take the Oracle.Database function and add my SQL statement?
I already know what I want, I don't want it to download all the table names. According to the help page, I should be able to do this but it does not provide a syntax example
Also, I don't understand what "optional options as nullable record" means.
Below is what function and arguments the help page notes. How do I use this?
Oracle.Database(server as text, optional options as nullable record) as table
Any help is greatly appreciated.
Thank you,
Jessica

When I try this, I get an error
DataSource.Error: Oracle: Sql.Database does not support the query option 'Query' with value '"Select * from Owner.View_Name"'. Details: null
I'm trying to download oracle data from a view into power query - Power Query navigator does not list th eviews from my source, it lists only the tables. When I try write sql statements, it throws me the above
error. This is what I tried
Oracle.Database("Source/Service",[Query="Select * from Owner.View_Name"])
Any ideas how to fix this?

'Merge' function in Power Query

Similar Messages

Maybe you are looking for