Curl command to parse web sites

Hi there,
I am trying to parse price details on the www.travelbag.co.uk web site. Basically I use a preformatted URL which contains date ranges and hotel information and put that into the Curl command. The idea then is via Applescript to add 1 to the day and have that running in a loop so I can determine which times of the year are the cheapest for a particular hotel etc. I have a working set up for the KenwoodTravel.co.uk web site but the problem with travelbag is that it first returns a holding page (like a please wait whilst we get the details). The Curl returns this http response and finishes. Doing it via a web browser works of course but I was wondering if anyone knew any tricks with the Curl command to wait for the full response?
Thanks,
Mike...
PS hope this all makes sense.

It is probably working through AJAX, much like the forums here. Your best bet is to debug the site in Safari's developer mode to try to find the AJAX request. You can then feed that into Curl.

Similar Messages

When I try to same an image under the command 'save as' it will only let me save as file types 'firefox document' or 'all files' and when I try to look at them later they dont work. How can I save an image as the same file type it is on the web site?!

When I try to save an image under the command 'save as' it will only let me save as file types 'firefox document' or 'all files' and when I try to look at them later they dont work. How can I save an image as the same file type it is on the web site?!
== This happened ==
Every time Firefox opened
== I updated to one of the firefox versions (Not sure which one it was)

Thanks Alex, but sadly I already tried that. Neither .docx or .xlsx files show up in the content list. They both show as a Chrome HTML document so changing how Firefox addresses those doesn't help since it thinks its the same type of file. I don't think I can manually add files into the "Content Type" left side nav.

Why has "Pin as App Tab" command disappeared when I right click on a web site?

Very much like FF4 and especially App Tabs. Today when I relaunched FF (1) App Tabs have disappeared (I have been reading how to keep them when I reopen FF) and (2) I no longer have a "Pin as App Tab" command when I right click on a web site I want to pin. Anyone got any clues? I am wondering if there is a conflict with Cooliris?

Start Firefox in [[Safe Mode]] to check if one of the add-ons is causing the problem (switch to the DEFAULT theme: Tools > Add-ons > Appearance/Themes).
* Don't make any changes on the Safe mode start window.
See:
* [[Troubleshooting extensions and themes]]
If it does work in Safe-mode then disable all extensions and then try to find which is causing it by enabling one at a time until the problem reappears.
* Use "Disable all add-ons" on the [[Safe mode]] start window to disable all extensions.
* Close and restart Firefox after each change via "File > Exit" (Mac: "Firefox > Quit"; Linux: "File > Quit")

How do I get Firefox to display more than one web site that I am developing using Localhost command?

I write the html code for the four web sites I develop and the one I maintain. I want to view the sites before I put them on the internet. I am able to do this with only one of the five. What, if anything, to I have to change in the browser to allow me to do this? I have edited the host file to include the names of the files on my computer of each of the sites. What should I do now?

You'd like to be able to access each site with its own locally-hosted url, correct? (for example- http://localhost1/, http://localhost2/ etc, )? That's a ''server'' setting, rather than a browser one.
How it works will depend on what software you are running to serve the pages. ''Apache'', for example, uses [http://httpd.apache.org/docs/2.2/vhosts/ Virtual Hosts] (and in this case, specifically [http://httpd.apache.org/docs/2.2/vhosts/name-based.html Name-based Virtual Hosts] ). There's usually an example in the configuration file to experiment with (be sure to back it up first).

What is missing when I get a blank white screen when I try to Access some web sites?

On some web sites, I get a blank white screen when I try to access them.

You can try these steps in case of issues with web pages:
You can reload web page(s) and bypass the cache to refresh possibly outdated or corrupted files.
*Hold down the Shift key and left-click the Reload button
*Press "Ctrl + F5" or press "Ctrl + Shift + R" (Windows,Linux)
*Press "Command + Shift + R" (Mac)
Clear the cache and remove cookies only from websites that cause problems.
"Clear the Cache":
*Firefox/Tools > Options > Advanced > Network > Cached Web Content: "Clear Now"
"Remove Cookies" from sites causing problems:
*Firefox/Tools > Options > Privacy > "Use custom settings for history" > Cookies: "Show Cookies"
Start Firefox in <u>[[Safe Mode|Safe Mode]]</u> to check if one of the extensions (Firefox/Tools > Add-ons > Extensions) or if hardware acceleration is causing the problem.
*Switch to the DEFAULT theme: Firefox/Tools > Add-ons > Appearance
*Do NOT click the Reset button on the Safe Mode start window
*https://support.mozilla.org/kb/Safe+Mode
*https://support.mozilla.org/kb/Troubleshooting+extensions+and+themes

Multiple Web Sites with Personal Domain Names - Overview

I have read through the last 10 pages of these discussions and have almost worked this out, but I need some help.
Problem:
I want to publish and edit two separate web sites with their own, individual, personal web addresses
from the same user account on the same Mac using iWeb '08 (2.0.2) and my .Mac account
_So far:_
I have purchased and registered the two domain names (with Cheap-DomainRegistration.com)
I have configured the CNAME to point to web.mac.com, and "Set up a Personal Domain" in iWeb successfully, but using a separate Mac for each web site.
The two sites work great.
_What (I think) I need to know:_
(excuse me for copying these posts out, but I can't find a way to link them to this post)
1) Should I be using iWebSites as suggested by Old Toad (posted Jan 24 in response to StAnNe's "Multiple Websites--HELP!!!")?
I use iWebSites to manage multiple sites.. It lets me create multiple sites and multiple domain files.
2) Should I be using Mireille's approach (also posted Jan 24 in response to StAnNe's "Multiple Websites--HELP!!!")?
Yes you are correct in the thought that with a family pack you can use different accounts and that is the easiest way to upload with one click to .mac. But it is still possible to publish differents sites to one .mac account if that is all one has. Even if the sites are in one domain file each purchased mysite.com domain name purchased can be pointed to a different page in the site
Look at it this way
Original Poster has
site1 page 1 page 2 and so on
then he/she has brothersite page 1 and so on.
They each have a domain name purchase wherever
then site1domain.com is forwarded to site 1 page1
and brotherdomain.com is forwarded to brothersite page 1.
Even though both sites are in the same file they do not have anything to do with each other.
This is one possibility there are others but for a novice user this could be the way to go for simplicity.
(Mireille, if you're there, can you clarify what you said - thank you)
3) Would I use Roddy's fix (posted Jan 23 in response to canadensis' "Publishing Multiple Websites?")?
Here's an example of how you can separate two websites that are on the same domain file.
Quit iWeb
Create a new folder on your desktop and call it "iWeb Sites".
Inside this folder create two more - Website A, Website B.
Go to Home Folder/Library/Application Support/iWeb and copy your domain file - command C
Paste this into folders A and B - command V - and also paste a copy of it somewhere else - like in Docs - in case you make a mistake!
Double click the domain file in the folder Website A - this will launch iWeb.
In the left column, delete site B, save and quit iWeb.
Double click the domain file in the folder Website B to launch iWeb.
Delete website A, save and quit.
Drop the iWeb sites folder into your Home Folder.
If you want quick access to this folder you can highlight it and do command L to create an alias to leave on the desktop.
From now on, to launch any site in iWeb you open its folder in the iWebsites folder and double click the domain file.
This is not necessary when you are working on only one site as iWeb saves the domain file of the last site you were working on to Home Folder/Library/Application Support/iWeb. When you open the iWeb application, the last site you worked on will be launched.
Summary:
I'm not sure if some of the answers in previous discussions allow for personal domain names, which is what I need.
I would be very grateful for any suggestions as I'm getting bogged down.
Many Thanks,
Jeff

When you said, "you don't need to do CNAME for both sites…", would this method still allow me to use personal web addresses for both sites?
Yes. With "Ordinary Forwarding" you normally just type your .Mac url (web.mac.com/username/sitename) into a form at the place where you have your name.
I thought I was using 'web.mac.com' as the 'www' CNAME (alias) for my personal domain name (web address), so that when someone typed in my personal domain name they would 'go' to the domain registration location, which would then pass it on to the .Mac server, where my web site is hosted.
That's exactly right. It's just not the only way to do that. Ordinary Forwarding is another way, but it differs in terms of what appears in the address bar of the browser. Either you will see web.mac.com/username/.... or, if you add "masking", you will see your personal name for all pages. The CNAME method results in a address bar that reads www.myname.com/sitename/pagename.html.
Am I way off?
All help gratefully received,
Jeff

Can't start the web site of Reports Services

Does anybody know how to fix this problem?
After installation of the 9iAS, can't start the web site of
Reports Services
and Form Services from the URL listed in the postinstallation
guide using
default port no 1950. The dev60html and devdem60/demo
directories are not
existing, the runrep.htm and runform.htm files are in the
directory
tools/web60/html. I also tried to replace the dev60html with
tools/web60/html
in the URL but it won't work.
During the installation the discoverer and portal tools failed,
the portal
hanged at 100%, and the discoverer didn't install the util
directory. But we
can login to the portal from the URL.
Below is the message when I start from the command line: I am
not sure if it is
working.
$ reports60_server start
Stopping Reports Server Rep60_abc.com processes...
kill -9 5647
Reports Server Rep60_abc.com stopped.
Starting new Reports Server Rep60_abc.com .
rwmts60 name=Rep60_abc.com
Stopping Reports Server Rep60_abc.com processes...
kill -9 5910
Reports Server Rep60_abc.com stopped.
$ reports60_server start
Stopping Reports Server Rep60_abc.com processes...
Could not find running instance of Reports Server
Rep60_abc.com
Starting new Reports Server Rep60_abc.com .
rwmts60 name=Rep60_abc.com
Thanks

Hi,
Dumb question : is one of your server listening on port 7001 ? If not, should it ?
What is the expected behavior ? A managed trying to connect to the admin server ?
Check the value for ADMIN_URL value, set in the startManagedServer.sh or cmd.
Another dumb question : do you have a firewall ?

When trying to send a comment to a web site I get a message, Firefox has blocked this site, how can I turn this off?

when trying to send a comment to discussion sites I get an error message that says "Firefox has blocked this site" actions offered "allow"
or "X". When I enter "allow" it ignores the command and the message disappears. A lot of wasted thought and typing. How do I turn this feature off.

See: Tools > Options > Advanced > General : Accessibility : [ ] "Warn me when web sites try to redirect or reload the page"
See: [[Options window - Advanced panel#General_tab]]
See also: http://kb.mozillazine.org/accessibility.blockautorefresh

How do I set up a root directory for a web site

So this is probably a naive newbie question.
I have been creating websites driven by php/mysql on my mac for years using the previous the web preference pane to run Apache and since that went away using XAMPP to provide Apache/php/mysql functionality on localhost.
I decided to download OSX server and try to use that. Figured might as well go native MAC.
I am baffled.
The manual says to set up a web site in Server by working through the Websites Services interface. I did that and pointed the setup at the directory where I have all of my web site files. I now have no idea how to access that page from my local machine.
I have tried:
http://localhost/myURL
http://localhost/~username/Sites/[root file directory] -- where the files are stored
I also tried some typical localhost ip address
None access my site.
How do I access my site over my local netework?
I feel like just dumping server.app and stting things up through the terminal.
Thanks,
--Kenoli

How familiar are you with the bash command line and with Apache, and how much do you want to learn about Apache and related topics? (If you're not already, then I'd encourage learning about this material, as this knowledge is very useful for maintaining and updating sites, and knowledge of correctly securing files and directories is critical to building and maintaining secure web sites — all of managing and running and updating a web server builds on these foundations. There are various good books and web sites on Apache, and the Apache web site has various materials available, if you don't already have something.)
The localhost host name and the 127.0.0.1 IPv4 address are both synonyms for the "me" host. It's the name and the address that always gets you to the current host.
With http://localhost/ or http://127.0.0.1 specification, what you enter at the URL address bar is relative to the root directory specified for the default web server, and will default to the index.html or index.php page or whatever the default web site is configured to use. With OS X Server, this root directory is configured via Server.app. To use Server.app, launch it, connect to the server, select Websites from the left navigation, and you (by default) will see a default site and a secure site. Select the default site. Select the edit button below the site to edit (and to view) the configuration.
With the details of the default site visible, you will see the path to the default web directory. This path to the default web directory is where the files for the default web sever location reside.
The default path to the default web site is /Library/Server/Web/Data/Sites/Default
If you place foo.html in that directory (or whatever the defailt web site is using), you can view it via your web browser and the URLs:
http://127.0.0.1/foo.html
http://localhost/foo.html
To get to the default web site using the GUI, launch Finder and select Go To Folder, and enter that string into the box. You will now get a GUI view of that folder. (I'd strongly recommend using the command line here, and not the GUI. To get to the directory from the command line and then list the files present, launch Terminal.app from Applications > Utilities folder and issue the following commands:
cd /Library/Server/Web/Data/Sites/Default
ls
If you want to test something in the browser, you can use the file:///path-to-file-to-test mechanism. Most web browsers will allow you to render local files directly using file and three slashes.
You will need to ensure the executable files (scripts) are marked as executable to OS X, using a Terminal.app command such as the following:
chmod o+x path-to-file
Enabling ~/user per-user sites involves directly editing the configuration file, as Apple has disabled that by default.
There's an Apache tutorial with an overview of what's involved with enabling per-user directories here, and there's what's been listed in my earlier replies and links. With OS X Server Mavericks, the default configuration file for Apache is located in the Library folder, and can be viewed with the following Terminal.app command:
cat /library/server/web/config/apache2/httpd_server_app.conf
This file is the OS X Server equivalent of the oft-cited httpd.conf file you'll see listed in various documentation.
If you want to enable the per-user directories or tweak other Apache settings, then shut down Apache from Server.app or from the command line, and then edit the configuration file with nano or some other command-line editor using a command similar to this:
sudo nano /library/server/web/config/apache2/httpd_server_app.conf
you'll end up finding and uncommenting (removing the #) from the following two lines:
#LoadModule apple_userdir_module ${SERVER_INSTALL_PATH_PREFIX}/usr/libexec/apache2/mod_userdir_apple.so
#Include /private/etc/apache2/extra/httpd-userdir.conf
If you have specific questions about this, then I and others here can answer those. If you don't understand something, then asking a question about that can help you learn, and can help me understand what it is that I'm posting in my replies here that's confusing you. There are also various resources available for learning Apache, and I'd encourage spending a little time with those if you've not already done so or feeling at all unsure about how that tool works; I end up re-re-reviewing the Apache docs fairly regularly.

HT1229 How do I move a photo from iPhoto to a web site where I'm trying to sell an item?

I want to post a picture on a web site where I'm trying to sell an item. I know that I must transfer the picture to another location in order to access it through the web cite, but I don't know how.

Either use the media broswer (on the left of the browser upload window unders media ==> photos ==> iPhoto)
or
to resize first use the export command - for details on accessing your files and on exporting see the user tip on each one
LN

Animation inside movie clip symbols in "header" part of web site

Question: i have to make an animated header for the web site. several seconds it's going on and is stoped with action script command stop(); but i need one elenment inside flash-header continue animating like here: http://ves-lab.ru/ a rolling logo in upper left corner of header. If somewhere the answer has already been written - it would be nice someone write a link to this source. Thanks.

Make that entity that continues to animate a separate movieClip with its own animation.

Some web sites not working in FF but they are OK in Safari.

A couple of times lately I have had web sites not work correctly with Firefox. I need to switch to Safari to complete the tasks I needed to do. My FF is running on Mountain Lion and is up to date.
Thanks,
MJT

Hello michaeljt, the sites you have problem, is it with a lot of flash content ?
Many site issues can be caused by corrupt cookies or cache. In order to try to fix these problems, the first step is to clear both cookies and the cache.
Note: ''This will temporarily log you out of all sites you're logged in to.''
To clear cache and cookies do the following:
#Go to Firefox > History > Clear recent history or (if no Firefox button is shown) go to Tools > Clear recent history.
#Under "Time range to clear", select "Everything".
#Now, click the arrow next to Details to toggle the Details list active.
#From the details list, check ''Cache'' and ''Cookies'' and uncheck everything else.
#Now click the ''Clear now'' button.
Further information can be found in the [[Clear your cache, history and other personal information in Firefox]] article.
Please reload the webpage while bypassing the cache using '''one''' of the following steps:
*Hold down the ''Shift'' key and click the ''Reload'' button with the left mouse key.
OR
*Press ''Ctrl'' + ''F5'' or ''Ctrl'' + ''Shift'' + ''R'' (Windows and Linux)
*Press ''Command'' + ''Shift'' + ''R'' (Mac)
See if this helps solve the issues you had and please report back to us!
thank you

Representing a web site in an indented list

Does any one have any ideas of how I can write a program which traverses a web site and writes its structure in an indented list text file format?

You will have to parse the input stream and build a tree and then convert the tree into an text file with desired formatting.
Something like:
Connect to a URL,
Get the stream,
parse and locate all href,
build list,
iterate through the list and repeat above steps
and you will have a tree in the end.
Hope this helps.
Let me know if you have questions.

How do i get my screen size of web sites back to normal size.

how do i get my screen size of web sites back to normal size.

Try COMMAND and two finger drag on the trackpad.
Ciao.

Solaris 10 11/06 - Cannot access external web sites from server

Hi,
I am relatively new with Solaris so I hope this is something simple. I just installed Solaris 10 11/06 with the `Secure By Default` option.
From the command line I can use dig and nslookup to resolve external web sites fine, but if I try to use the update conection manager (desktop), mozilla browser (desktop), or pkg-get (command line) it fails to resolve the external sites. I have a valid resolv.conf file and I have named installed and running.
It is on a LAN with other servers that have no problems with external access. I am running as root in all instances, so is there some permissions issue I am missing with this install/release?
I don't want to open up the install more than I have to.
Thanks,

Hi again,
I figured it out. The nsswitch.conf file was not referencing dns as part of the `hosts` and ipnodes` resolution methods. Once I added this and restarted named it worked.
group: files
hosts: files dns
ipnodes: files dns
networks: files
-----------------------

Curl command to parse web sites

Similar Messages

Maybe you are looking for