How to automate/parameterize/loop power query?

Question

Is it possible to write a single generic powerquery command that'll loop through a list of URLs and load the data to a single table? the webpages in question have the same format & table names, just the data contained within is different. I verified
this by manually defining a powerquery from web import, importing, changing the url, and hitting refresh
Is it possible to write a generic powerquery import to pull the data in and have it loop to append the data to a target worksheet?
I tried macro recording some of the powerquery actions of defining an import but it didn't record anything so im assuming i cant use a vba loop to do this.
I want to avoid having to manually define each from web import explicitly. There's a few hundred i'd need to do.
Jakub @ Adelaide, Australia

Answer 1

Here's a quick walkthrough for how to do something like this. I recommend reading through to the end before trying this, because there will a step later on which you'll actually want to do first to simplify things.
Let's say I want to get presidential election results for multiple US States from Wikipedia. So to start, I use the "From Web" option inside Power Query and specify an URL of
http://en.wikipedia.org/wiki/California. This lets me pick between several sections of the web page, and I choose the one called "Presidential elections results" and edit it. The results look good
in the preview, so I go to the "advanced editing" view that shows me the M query:
let
    Source = Web.Page(Web.Contents("http://en.wikipedia.org/wiki/California")),
    Data5 = Source{5}[Data],
    ChangedType = Table.TransformColumnTypes(Data5,{{"Year", type number}, {"Republican", type text}, {"Democratic", type text}})
in
    ChangedType
The numeric index "5" worries me a little because I suspect that there's not enough regularity in the Wikipedia pages for the US states for the 6th section to always be the one that I want. So I edit the query to index by the section name instead
and confirm that it still produces the results I want:
let
    Source = Web.Page(Web.Contents("http://en.wikipedia.org/wiki/California")),
    Data5 = Source{[Caption="Presidential elections results"]}[Data],
    ChangedType = Table.TransformColumnTypes(Data5,{{"Year", type number}, {"Republican", type text}, {"Democratic", type text}})
in
    ChangedType
Now I need to turn this into a function that can be applied to multiple pages. The only way to do that is to edit the M query directly, but the transformation is relatively straightforward. Again, I confirm that the output is still what I expected after
making these changes.
let
    Election.Results = (state) => let
        Source = Web.Page(Web.Contents("http://en.wikipedia.org/wiki/" & state)),
        Data5 = Source{[Caption="Presidential elections results"]}[Data],
        ChangedType = Table.TransformColumnTypes(Data5,{{"Year", type number}, {"Republican", type text}, {"Democratic", type text}})
    in
        ChangedType
in
    Election.Results("California")
Now, I want to apply this function to multiple web pages. These might come from a different Excel sheet or from an external file, but I'm just going to build a table of web pages manually. I'll keep the function around, but won't use it just yet in the overall
formula. Once more, I confirm that the output is what I expect, which is a single table listing three states.
let
    Election.Results = (state) => let
        Source = Web.Page(Web.Contents("http://en.wikipedia.org/wiki/" & state)),
        Data5 = Source{[Caption="Presidential elections results"]}[Data],
        ChangedType = Table.TransformColumnTypes(Data5,{{"Year", type number}, {"Republican", type text}, {"Democratic", type text}})
    in
        ChangedType,
    States = Table.FromRows({{"California"}, {"Idaho"}, {"Massachusetts"}}, {"State"})
in
    States
Now I can use the "Insert Custom Column" feature of the UI to apply my function to the State column. This looks like "=Election.Results([State])" in the dialog. But when I click OK, something unpleasant happens: I get prompted several
times for "data privacy" settings for each of the URLs I'm trying to access. This is quite annoying, and we're looking for ways to improve the experience -- perhaps by letting you pick that the privacy settings apply to all of
http://en.wikipedia.org and not just to the specific URL you're trying to access.
Meanwhile, you can achieve the same thing manually with a kind of underhanded trick by creating a throwaway query which references
http://en.wikipedia.org and one other website, and using the "data privacy" prompt from that query to define a privacy rule for the entire domain. Exit the editor and create a new "Blank Query" with
this single formula: "= Binary.Length(Web.Contents("http://en.wikipedia.org")) + Binary.Length(Web.Contents("http://www.bing.com"))" This will prompt
you to pick a privacy setting for those two websites (I would suggest that "Public" is the right choice). Once you've done that, you can discard this query; its work is done.
Now if you go back and repeat the steps for the "elections" query, you'll avoid the privacy prompt.
At this point, you'll have a table with two columns: one that has the state, and one that has the results as Table values. We want to expand out those values, so we click on the expansion indicator in the column header for Custom and pick all fields. The
result is a table with four columns: State, Custom.Year, Custom.Republican and Custom.Democratic. The latter three fields can be renamed from either the UI or by editing the M code. In my case, I ended up with a program which looks like this:
let
    Election.Results = (state) => let
        Source = Web.Page(Web.Contents("http://en.wikipedia.org/wiki/" & state)),
        Data5 = Source{[Caption="Presidential elections results"]}[Data],
        ChangedType = Table.TransformColumnTypes(Data5,{{"Year", type number}, {"Republican", type text}, {"Democratic", type text}})
    in
        ChangedType,
    States = Table.FromRows({{"California"}, {"Idaho"}, {"Massachusetts"}}, {"State"}),
    InsertedCustom = Table.AddColumn(States, "Custom", each Election.Results([State])),
    #"Expand Custom" = Table.ExpandTableColumn(InsertedCustom, "Custom", {"Year", "Republican", "Democratic"}, {"Year", "Republican", "Democratic"})
in
    #"Expand Custom"
And the job is done!
Actually, the hard part might just be getting started. Finding a set of web sites which is consistent enough for a single parsing function to work can be rare. As an example, when I tried to add "Washington_(U.S._state)" to my list of states, I
discovered that its table was formatted differently than the others. And when I tried to add "New_york_state" to the list of states, I found that its section was named "New York State Presidential Election Results" instead of just "Presidential
election results". So it's possible that you'll have to do an awful lot of tweaking.
Nonetheless, this should cover the basic steps and help you get started.