Using automator to download list of URLs
I have a list of URLs that I want to use automator to download. The URLs link to PDF files on a website.
I am using the following workflow:
"Get Specified Text (putting the list of URLs in here)" -> "Filter paragraphs (filtering out paragraphs beginning http://)" -> "Download URLs".
The URLs are getting passed and filtered correctly but the files that are downloaded are generic html files (no specific name, just a number) and not the PDFs. I can paste each URL into the browser individually and it will download the PDF so I know the URLs are correct. How do I get it to download the PDFs instead of these html files?
Message was edited by: bercg
Is there something else on the web page, such as javascript or formatting around the PDF? You might try getting the links directly with something like:
1) *Get Specified URLs*
2) *Get Link URLs from Webpages*
3) *Filter URLs* -- filter for just the PDF file links
4) *Download URLs*
Similar Messages
-
With Mac Defender the Safe Downloads List has become better know. However, I do have a question about it. All the descriptions on Apple's web site about this, naturally, speak of when downloading using Safari. Like a number of others I like to use FireFox and Google Chrome. Is anything done in either of these browsers to use the Safe Downloads List to protect us?
Unless, it is implemented in some OS feature used by all browsers to download and open files, this is not Apple's responsibility. If it is up to the browser to deal with this, I want to encourage those browser providers to add this feature to the Mac OSX implementation.
Thanks,
Brucethe defintions of Apple add MD5/SHA checksums to the Installer feature of OSX directly, so bad apps / malware is being fingerprinted as "block them". The list contains various versions of MacDefender as well as a handful others.
So say, if Macdefender version xyz find its way onto your OSX ( regardless which Browser has let it slipped through ) it can not install itseld, it will be blocked by OSX itself to launch its installation mechanism.
Vice versa this means : every installation package you launch is first being scanned by OSX if it matches the present fingerprints of the bad apps. And only if it is given the "clean" status, the actual install process is beginning.
It is the same way like traditional Antivirus Defintions work. Blacklist and Whitelist. Of course this is a cat and mouse race between the Software Vendors and the criminals. Apple will include every known fingerprint into their defintions and push them out to the OSX boxes daily, but therefore at least one known new bad app has to be known ( at least one machine needs to be infected to know the checksum of the newer installer and to include its checksum to the updates ..)
Yes the web is a ugly place.. no OS is safe from those threats these days. Would Linux see a rise in usage, they would target that too. -
I have the download list for firefox but accidentally deleted the file on my computer. I need the file and was wondering if there was a way of using the download list to access the file somehow, I don't think the file os on the internet either as I haven't found it
Hello!<br>
Go to the downloads window, right-click, and click "copy download link" or "go to download page". You should then be able to re-download it only if it's still in the downloads window -
im new at this lol just wondering if anyone knew how to delete something in ur download list if u dont wanna download it?
Deleting the download will not stop it from using your "credit" in iTunes. The content has already been purchased.
You could simply let it download or delete the entry from the downloads page. -
I want to assign the Url of a Sharepoint list page to a Constant, and I am using it into User Control navigate Url.
But it is Not working there , Could you Please Suggest me the Better way..
Regards
NiteshYour question isn't clear. Can you explain what you're trying to do and what you've attempted?
-
Fist, I should mention I am not a programmer but am eagerly learning powershell!
I am looking for an automated solution to accomplish what I am currently doing manually. I need a script that would combine the following:
Reach out to a list of websites (probably a loop of some sort since the list will come out of a spreadsheet which could contain 1 or 100 different sites)
Search each page for a specific word or words (not contained in the spreadsheet though that may make it more scalable)
Save the URL of the site(s) that contained the keywords to one text file (versus the multiple .html files I am creating today)
Have the output contain which words it found on which site.
If not overly complicated, I would like to schedule this to recur once a week.
A working script would be ideal, but even the resources that show me how to incorporate each element would suffice.
I have had success pulling down the full content of the listed pages and saving them to a directory, which requires manual intervention.
So far this works, but it's not scalable:
Set-ExecutionPolicy RemoteSigned
$web = New-Object Net.WebClient
$web.DownloadString("http://sosomesite/54321.com") | Out-File "C:\savestuffhere\54321.html"
$web.DownloadString("http://sosomesite/54321.com") | Out-File "C:\savestuffhere\65432.html"
Get-ChildItem -Path "C:\savestuffhere\" -Include *.html -Recurse | Select-String -Pattern "Keyword 1"
In otherwords, I have to manually replace the "http://sosomesite/54321.com" and "C:\savestuffhere\54321.html" when the URL changes to .\65432.com and the output name to match. That works fine when it's a couple sites, but again,
is not scalable.
Then, to see if any of the saved file's contain the keyword(s), I have to search the directory for the keyword which I am using:
Get-ChildItem -Path "C:\savestuffhere\54321.html" -Include *.html -Recurse | Select-String -Pattern "Keyword 1"Hi Sure-man,
Sorry for the delay reply.
To automatically Reach out to all urls, you can list all urls in a txt file "d:\urls.txt" like this:
http://sosomesite/54321.com
http://sosomesite/65432.com
Then please try the script below to save the URL of the site(s) that contained the keywords to one text file "d:\outputurls.txt":
$urls = get-content d:\urls.txt
foreach($url in $urls){
$results = $web.DownloadString("$url")
$matches = $results | Select-String -Pattern "keyword1","keyword2"
#Extract the text of the messages, which are contained in segments that look like keyword1 or keyword2.
if ($matches.Matches){
$Object = New-Object PSObject
$Object | add-member Noteproperty keyword $matches.Matches.value
$Object | add-member Noteproperty URL $url
$output+=$Object}
$output|Out-File d:\outputurls.txt
If you want to schduled this script in Task Scheduler once a week, please save the script above as .ps1 file, and follow this article:
Weekend Scripter: Use the Windows Task Scheduler to Run a Windows PowerShell Script
If I have any misunderstanding, please let me know.
I hope this helps. -
Using Automator for software updates
Hi all. I apologize in advance if this has been asked and answered numerous times; I did several searches using different terms and did not find my answer.
Can I use Automator to do all my software updates completely automatically? Including doing the restarts? I know this is a very basic question for most all of you; I can use my computer but really don't know a lot about using it to the max. I would like to be able to have all the software updates done during the night, while the computer is unattended.
Thanks.If you are not too keen on running those commands you might want to try http://www.igetter.net/iGetter.html This download manager seems to allow the scheduled downloading that you need. I think you would have to go to the Apple site and find the url for the standalone updaters to find the link after Software Update identifies them. Then use this program to schedule the downloads. Sometimes those seem more stable anyway.
The program us shareware. It appears from the description that the program remains fully functional but puts up a nag screen if the program has not been registered. -
Hello.. I have one problem. I used automator to automaticly sync my photos to iphoto. Now I don't want it any more. How can I disable it?
If you have iPhoto v9.6.1 then just launch it. Done.
IF you don't, download it:
Go to the App Store and check out the Purchases List. If iPhoto is there then it will be v9.6.1
If it is there, then drag your existing iPhoto app (not the library, just the app) to the trash
Install the App from the App Store.
Sometimes iPhoto is not visible on the Purchases List. it may be hidden. See this article for details on how to unhide it.
http://support.apple.com/kb/HT4928
One question often asked: Will I lose my Photos if I reinstall?
iPhoto the application and the iPhoto Library are two different parts of the iPhoto programme. So, reinstalling the app should not affect the Library. BUT you should always have a back up before doing this kind of work. Always. -
I want to save images to memory 50 images
My software processing ----
1. Download Image from URL every 60 secs and all downloaded images have time to use 15 mins
2. When download image done save images to memory
3. Then show images in picturebox
4. When user click button get next image form memory and delete old image from memory
5. If what images not use in 15 mins auto delete from memory
Help me ... Thank you a lotOk chechk the below method/Events that will download 50 image and fill a List<image> object into memory, you need to add a picturebox1 and button1 to your form, I used my profile pic here on msdn the link you've send didn't work:
//Global Variables
List<Image> li = new List<Image>();
int second = 0; //Form Initaile
public Captcha()
InitializeComponent();
//Async download event
private void ReadCallback(IAsyncResult asynchronousResult)
HttpWebRequest request = (HttpWebRequest)asynchronousResult.AsyncState;
HttpWebResponse response = (HttpWebResponse)request.EndGetResponse(asynchronousResult);
using (StreamReader streamReader1 = new StreamReader(response.GetResponseStream()))
li.Add(Image.FromStream(streamReader1.BaseStream));
pictureBox1.Image = li[0];
//Download Method for 50 images
private void DownLoadImages()
try
for (int i = 0; i <= 50; i++)
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(new Uri("https://i1.social.s-msft.com/profile/u/avatar.jpg?displayname=fouad%20roumieh&size=extralarge&version=00000000-0000-0000-0000-000000000000"));
request.BeginGetResponse(new AsyncCallback(ReadCallback), request);
catch (WebException ex)
//Form Load
private void Captcha_Load(object sender, EventArgs e)
DownLoadImages();
And here on button click event we remove element and show the next one into picture box, and if used up all image we call the async download messages again to get a new list of images:
private void button1_Click(object sender, EventArgs e)
li.RemoveAt(0);
if (li.Count > 0)
{ //Show next
pictureBox1.Image = li[0];
else
DownLoadImages();
Also here a timer event I didn't enable but you can enable to check for the 15 mins and destroy the images list if passed:
private void timer1_Tick(object sender, EventArgs e)
second = second + 1;
int minutes = second / 60;
if (minutes >= 15)
li.Clear();
//Call downloadimages or else...
Fouad Roumieh -
Hello all!
I would like to be able to reply to a large amount of Craigslist emails by pulling the contact info from Craigslist using Automator and putting the emails in a text document.
I have Automator set to:
1. Open Safari
2. Open the Craigslist section an city I want.
3. Pull a list of active listings URL'S
Beyond that, I can not figure out how to have Automator open one page at a time and click the contact info button and then copy the needed info into my text document.
I can have it open all 150 links at one time.....lol but that wont work on my Macbook!!!!
Help please!!!Hello
You may try the following AppleScript script. It will ask you to choose a root folder where to start searching for *.map files and then create a CSV file named "out.csv" on desktop which you may import to Excel.
set f to (choose folder with prompt "Choose the root folder to start searching")'s POSIX path
if f ends with "/" then set f to f's text 1 thru -2
do shell script "/usr/bin/perl -CSDA -w <<'EOF' - " & f's quoted form & " > ~/Desktop/out.csv
use strict;
use open IN => ':crlf';
chdir $ARGV[0] or die qq($!);
local $/ = qq(\\0);
my @ff = map {chomp; $_} qx(find . -type f -iname '*.map' -print0);
local $/ = qq(\\n);
# CSV spec
# - record separator is CRLF
# - field separator is comma
# - every field is quoted
# - text encoding is UTF-8
local $\\ = qq(\\015\\012); # CRLF
local $, = qq(,); # COMMA
# print column header row
my @dd = ('column 1', 'column 2', 'column 3', 'column 4', 'column 5', 'column 6');
print map { s/\"/\"\"/og; qq(\").$_.qq(\"); } @dd;
# print data row per each file
while (@ff) {
my $f = shift @ff; # file path
if ( ! open(IN, '<', $f) ) {
warn qq(Failed to open $f: $!);
next;
$f =~ s%^.*/%%og; # file name
@dd = ('', $f, '', '', '', '');
while (<IN>) {
chomp;
$dd[0] = \"$2/$1/$3\" if m%Link Time\\s+=\\s+([0-9]{2})/([0-9]{2})/([0-9]{4})%o;
($dd[2] = $1) =~ s/ //g if m/([0-9 ]+)\\s+bytes of CODE\\s/o;
($dd[3] = $1) =~ s/ //g if m/([0-9 ]+)\\s+bytes of DATA\\s/o;
($dd[4] = $1) =~ s/ //g if m/([0-9 ]+)\\s+bytes of XDATA\\s/o;
($dd[5] = $1) =~ s/ //g if m/([0-9 ]+)\\s+bytes of FARCODE\\s/o;
last unless grep { /^$/ } @dd;
close IN;
print map { s/\"/\"\"/og; qq(\").$_.qq(\"); } @dd;
EOF
Hope this may help,
H -
Download multiple file URL's at once in Safari 5.1?
If I have a list of URL's of files (MP3's in this case) I used to be able to copy and paste the whole list into the Downloads window in the previous version of Safari and it would begin downloading all of them automatically.
With the elimination of the Downloads window in Safari 5.1 I can't figure out how to do this; I know I can download one URL at a time by pasting it into the address bar and Option-Return, but that doesn't work with a long list of URLs.
Any idea of how this works in Safari 5.1?
Thanks,
DaveProblem solved, thanks to your suggestion. I had installed third party software, Speed Download, which in previous versions of Safari would add a menu bar item allowing it to manage downloads. SD did not appear to work with Safari 5.1, the menu bar no longer allowed access to the software, but it was interecepting download commands (not opening the SD program to download them, however). I used the SD uninstaller to remove the program, and now Safari downloads behave normally. I did try safe mode boot, which did not solve the problem, but it was definitely the download manager causing the problem.
Thank you! -
Is there any way to recover my Downloads list? I hit "Clear List" by mistake.
Since your downloads went to a temporary folder, and there are many things that help you clean your system of "unwanted" files they may be gone.
In that case you may find the pages you got them from in your history (Ctrl+Shift+H") especially if they came from one site, or you know when you downloaded them.
The AwesomeBar search using strings from title and/or url that you use in the Location Bar also works in in your History & Bookmarks sidebars and Library List. Internal searches from the Location bar have ''[http://kb.mozillazine.org/Location_Bar_search#Location_Bar_search_.28internal_-_Auto_Complete.29 modifiers]'' as well to help you, but in your case you probably need to work with the Library Lists. -
JWS Download Dialog's URL(From:) different from browser's URL
Hi everyone,
My problem:
I have deployed application A.jar at: {color:#0000ff}192.168.0.10{color}
B.jar at:{color:#ff0000} 192.168.0.20{color}
they were alaway launched successfully.
but sometimes when I input {color:#0000ff}192.168.0.10{color} in the browser address, B.jar from {color:#ff0000}192.168.0.20{color} has be launched(*A.jar* has not),
when I clear the JAVA cache , the problem will not happen 。 but After a few days , the problem happened again...
i don't know how to slove it.
anyone can help me ? thanks
my english is poor ,i hope you will get my point.
, :)Hi!
You might have seen this post in the Automator part
of the forum... but I might have posted it in the
wrong place (got no interest:( )
I've managed to get automator to retrieve (and save
to hdd) an .m3u playlist from a radio's website that
broadcasts a program I like in mp3.
inside this m3u playlist is the link to an actual mp3
file... both the m3u and the mp3 change every week.
If I drag the m3u to TextEdit it opens up and I can
see the 3 lines of "source code" of which the last
line is the full url of the mp3 file (http://www
.thatradiosurl.com/audio/show-shownumber.mp3)
You already got the playlist into a text file, you have two options:
1. use Automator's 'Get Link URLs from Webpages' action to extract the URL. 'Get Link URLs from Webpages' is under Safari lib.
2. run Automator Python script from 'Get Link URLs from Webpages' in AppleScript. The script below with get your mp3 URLs:
set playlist_file to "/path/to/playlist/file.txt" -- change to your actual playlist file path
set _URLs to do shell script "/System/Library/Automator/Get\\ Link\\ URLs\\ from\\ Webpages.action/Contents/Resources/links file://" & quoted form of playlist_file
set mp3_URLs to ""
repeat with _URL in paragraphs of _URLs
if _URL contains ".mp3" then set mp3_URLs to mp3_URLs & _URL & (ASCII character 10)
end repeat
mp3_URLs is the list you want you extract. -
When using the camera downloader in Adobe Bridge CS6 with Nikon D5200 we are unable to see previews of the photos and it is very slow to download. The issue occurs under a the users rights, but not under admin level. This is a new issue.
Hi Jdentremont,
Lync client gets user photos by first querying the Address Book Web Query (ABWQ) service on the server, which is exposed through the Distribution List Expansion web service. The client receives
the image file and then copies it to the user's cache to avoid downloading the image each time it needs to be displayed. The attribute values returned from the query are also stored in the cached Address Book Service entry for the user. The Address Book Service
deletes all cached images every 24 hours, which means that it can take up to 24 hours for new user images to be updated in the cache on the server.
To troubleshoot your problem, please follow the steps below:
1. Navigate to
“X:\share\1-WebServices-1\ABfiles\000000000\000000000” folder. (ABS file share)
You should see some photo files in this folder as the following screenshot.
2. Delete all the files in this folder.
3. On test PC, delete local cache files.
%userprofile%\AppData\Local\Microsoft\Office\15.0\Lync\[email protected]
4. Sign-in Lync with the test account.
5. Go back to the ABS file share, check if there is any Photo file in the folder.
Best regards,
Eric
Please remember to mark the replies as answers if they help, and unmark the answers if they provide no help. If you have feedback for TechNet Support, contact [email protected] -
Unable to delete a file from the downloads list of...
i am an e7 user. I was recently downloading a couple of videos at the same time when my phone hanged and had to be restarted. When it did restart, most of the files had to be redownloaded. All except one. The downloads list claims that the file has been saved. The video plays bt isnt completely downloaded.
The problem now is that i cant delete the video through the file manager. The error msg says web:already in use.. When i tried deleting the file from the downloads list of the web browser, there was a similar msg.
So how do i remove this file from my phone??
I tried downloading a couple of file managing softwares bt that didnt really help....
Solved!
Go to Solution.Retry after clearing the Browser Cache.. If that does not solve.. Reboot the phone and try again...
Maybe you are looking for
-
My ipad2 is not restarting after updating to iOS7, I now only get the apple symbol? Please advise what I need to do to get it working again. Thanks.
-
Need help with pre ordering the iphone 4s.
I did everything and im at the screen where it tells you how much it is total and everything you chose. But it says No text plan, it didnt ask me about a text plan, is this something i have to do with my carrier to get a text plan?
-
Upgrade an iOS app made in Flash CS5 to Flash CS5.5
Hi, I developed an App and I published it on the Apple App Store about a month ago, when I was still using Adobe Flash CS5. Now I bought the latest version of Adobe Flash, the CS5.5 and I want to upgrade my exiting app with the Retina Display support
-
IPhoto '08 corrupting edited photos
Hello, all. I'm having an issue w/ iPhoto '08. Around 75% of the time when I edit a photo, iPhoto corrupts the edited picture. The thumbnail appears as you expect it, but when you go to look at the pic in full-screen mode, nothing is there. You don't
-
Kernal Panic Help with Macbook Pro 2010
Hello fellow Kernal Panic enthusiasts. I've had random panics more frequently on a Macbook Pro 2010. I've tried correcting permissions, resetting PRAM etc. It seem to happen at random, sometimes with flash (which has been disabled by default) in Chro