Retreiving HTML data from an URL

It is possible to get HTML data (HTML source) from Java (text), and store them in a string ? Using java application (no gui).

import java.net.*;
import java.io.*;
public class GetHTML
     public static void main(String[] args)
                if (args.length < 1)
               System.out.println("Usage: <URL>");
               return;
                try{
          URL url = new URL(args[0]);
                BufferedReader in = new BufferedReader(new InputStreamReader(url.openStream()));
                String line = "";
                while ((line = in.readLine()) != null)
                    System.out.println(line);
                }catch(MalformedURLException mue){
                    System.err.println("Bad URL: " + mue);
                    return;
                catch(IOException ioe){

Similar Messages

  • Fetch data from another URL

    Hi all,
    i need the code that can fetch the data from another URL like i need the new from different site and put into my database and then website useing the jsp or u tell me in which and how.
    thanks in advance.. URGENT
    Waiting................................................ UR rpely..................

    Hi,
    People generally ignore "URGENT" pleas on this forum... it's only urgent to you.
    You probably want to do a search for "web scraping"... basically parsing HTML pages as XML, and extracting the results.
    Recently dcminter posted a link to a very good article from the IBM website, which you should read.
    http://www-128.ibm.com/developerworks/java/library/j-jtp03225.html
    regards,
    Owen

  • Reading JSON data from a URL

    Hi all,
    I have a requirement of reading JSON data from a particular URL and using one of the value to set one property in iView. I need some info on how to get JSON data from a URL and extracting attribute's value from it.
    What are the APIs that can be used for this?Can anyone provide a solution/working example in Java for this?
    I am working on EP 7.3.

    Hi Tarun,
    JAXB should work for you. Take a look at this example:
    JAXB JSON Example | Examples Java Code Geeks
    Regards,
    Tobias

  • How to use the readLine()  method when reading data from a URL?

    Hello,
    I have a URL which contains text input.
    The only way to get the data from this URL is by opening a URL connection to it.
    I would like to get the data from this URL but at the same time I would like to be able to use readLine() method of BufferedReader in order to read the data line by line.
    My question is how do I combine between these two requirements in order to reed the data from the URL?
    Roy

    Hello Roy,
    can you try out this code.
      URL yahoo = new URL("http://www.yahoo.com/");
            URLConnection yc = yahoo.openConnection();
            BufferedReader in = new BufferedReader(
                                    new InputStreamReader(
                                    yc.getInputStream()));
            String inputLine;
            while ((inputLine = in.readLine()) != null)
                System.out.println(inputLine);
            in.close();
    Regards,
    Mohan R

  • Receive POST data from another URL and process using Struts

    Hi there
    We have a website and as of now we are receiving some data from another URL which is received as an appended part of the URL. My application uses Struts and we process the received data and send back a response.
    Now my question is, I have been asked to change this behavior because there are more parameters now which cannot be passed through the URL. I am supposed to get the data from the other URL as POST data (as a form) and I have to create a new Struts action to receive this data, process it and send the response back to the requesting URL.
    Please explain me how to do this using some example code snippets.
    Thanks a lot

    Lookup in XI is used to call the target data storage system and get data from there to your mapping programme.
    In XI you can do Lookup in Message Mapping, Java Mapping and in XSLT Mapping. Previously Lookup in XI was system dependent. But now what ever the system are i.e. SAP system or non-sap system(Oracle,MS SQL etc) lookup API are same.
    Overview of Lookup
    - Lookups are used to identify/request the data from mapping program.
    - It interrupt the process and looking for data which was stored in target system.
    - It get that data and comeback to process and continue with that data.
    Types of Lookups in XI
    - JDBC Lookup: JDBC lookup is used for accessing data from database (non SAP).
    - RFC Lookup: RFC lookup is used for accessing the SAP Data.
    - SOAP Lookup: SOAP lookup is used for accessing data from Webservice
    Steps to perform Lookup in Mapping
    Import package com.sap.aii.mapping.lookup.*;
    Create connection to the target Database system.
    // Determine communication channel created in ID
    Channel channel = null;
    channel = LookupService.getChannel("DB-SYSTEM-NAME","DB-CHANNEL-NAME");
    // Get system accessor for the channel.
    DataBaseAccessor accessor = null;
    accessor = LookupService.getDataBaseAccessor(channel);
    Build the Query String.
    Getting Result
    // Execute Query and get the values.
    DataBaseResult resultSet = null;
    resultSet = accessor.execute(Query);

  • Pull data from a table in HTML format from external URL into a Servlet

    How do I pull data from existing website html table into a Servlet?
    I will need to pull the data into a servlet. So, the servelet will need to go to the URL and get the data and save them as variables.
    I know I will need an array to store the data. I just dont know how to call the URL and have the servelet search the site for the <table> </table>.
    I will later use this to save to a database.
    Thanks
    Edited by: DJMegabit on Apr 14, 2010 7:36 PM

    Maybe give us more details about what exactly You'd like to achieve. I guess that there might be better solutions to the problem.

  • How to extract data from web URL

    I was doing one project which need to extract data from web pages and then analyze these data. the question is how to extract data from there, using html parser? need help, thanks a lot

    I was doing one project which need to extract data
    from web pages and then analyze these data. the
    question is how to extract data from there, using
    html parser? need help, thanks a lotTry this:
    http://java.sun.com/docs/books/tutorial/networking/urls/readingURL.html
    Or, like you said yourself, use an HTML parser:
    http://java-source.net/open-source/html-parsers

  • Reading binary data from a URL

    Below are 2 snippets of code that read data from a binary file. A small sample of the output is shown at the bottom of each code fragment. The first one uses the URL class to read a remote file. The ouput for this fragment is incorrect in some cases. The second uses a stream reader to read a local file (the output is correct in this case). Both fragments read the same file.
    As you can notice, some of the bytes read are the same in both cases. Some others are not.
    I hope you can suggest how I can fix my URL version.
    Thanks for your help.
    Miguel
    Program A: Reads a remote file with URL
    URL url = new URL("http:localhost//myfile.awg");
    URLConnection connection = url.openConnection();
    BufferedReader bin = new BufferedReader(new InputStreamReader(connection.getInputStream()));
    while(true) {
    System.out.println(Integer.toHexString(bin.read()));
    7d 3 0 0
    22 30 3 3e
    fd b9 2a 0
    b2 2 0 1
    Program B: Rads a local file
    DataInputStream din = new DataInputStream(new BufferedInputStream
    (new FileInputStream("c:\\inetpub\\wwwroot\\myfile.awg")));
    while(true) {
    System.out.println(Integer.toHexString(din.read()));
    8e 3 0 0
    99 30 3 3e
    81 b9 2a 0
    b2 2 0 1

    What can I use instead?
    Thanks,
    MiguelBufferedInputStream as in Program B.

  • Get all html files from a url

    Hi I am trying to do a website copyer and i don't know how get all the html from a url.
    By example:
    I want get all files html from http://forum.java.com/
    aAnybody has an idea of how to do this?
    Thanks

    Not easy. First, you'll have to look at networking programming - how to make URL connections etc.. Then, you'll have to look at HTML parsers - how to extract links, images, etc. from an HTML page. Also, you'll have to know how to do file I/O in order to save the HTML files obtained.
    I don't know of any Java-based web site copier open source projects that you can refer to, but you can take a look at the source code for projects such as HttpUnit and HtmlUnit. They contain code that'll help you with your web site copier.
    Good luck!

  • Retreive External data from SQL making connection to SQL server take a long time

    I have an Excel pivottable based on a view from SQL server. The view gets data from 2 databases on the sames server. I have the sames login for both databases and also the same rights (datareader, datawriter and owner). When running the view from within
    the SQL Server management tool it goes quick. In Excel once the connection is made everything goes fast. Only making the connection takes a very long time. When I use a view with data from a single database then everything is fine and goes fast.
    Example of the situation: I use the example view below in an excel pivot. This gives as result that the connection takes very long.
    SQLServer A
    View CustomerInfo which contains the following query.
    Select * FROM databaseA.dbo.Customers as Customers
    CROSS APPLY
    SELECT *
    FROM databaseB.dbo.Contacts as Contacts
    Where Customers.Number = Contacts.CustomerNumber
    ) AS Contactdata
    When I remove the CROSS APPLY part, the connection is made directly
    We login to SQL with normal logon  (not windows security) The user has the same rights on both databases.
    I hope the problem is clear. Does anyone have an idea of how I can solve this problem?

    -1* ( prior dataid ) = parentid
    is it multiplying -1 times the prior dataid ?that's exactly what it's doing
    you need:
    prior dataid = ABS(parentid);
    not knowing you're data, and therefore assuming that dataid is always positive (since it doesn't have an ABS), and that parentid can be positive or negative, your criteria becomes
    parentid = prior dataid (handles both are positive)
    or parentid = negative of prior dataid (handles parentid is neg, dataid is pos)
    since the "prior" must be right next to the column (dataid), I put parens around the whole thing, and then mult by -1

  • Retreiving a date from database and converting it into a string in a format

    hi all,
    iam having a date field which stores values ,for eg:2002-04-26 00:00:00.i want to retreive the date and i want to convert in a String in dd/mm/yyyy format.how can i do so.plz explain me..........thanks in advance

    If the field is a Date in your database, you can use the ResultSet.getDate() method, which will return a java.sql.Date (which extends java.util.Date).
    You can then use SimpleDateFormat to format it the way you like.
    See the API doc for more details.

  • How do I grab data from a url?query using LabView 8.0?

    If I copy the following url in my web browser, it will give me the quote of BBD-A.TO symbol from yahoo. The data is comma separated (csv).
    http://finance.yahoo.com/d/quotes.csv?s=BBD-A.TO&d​=t&f=sl1d1t1c1ohgvj1pp2wern
    I would like to know how to grab that information using LabView. I tried using datasocket VIs but I was unsuccessful, it seems that as soon as I put the query part of the url, I am not able to read (timeout occur).
    I think there might be another way to do it using .NET "httprequest" and "httpresponse" classes but again I was not successful. 
    I am really new to this, thanks for your help.

    Hi Ben,
          Try adding "[text]" to the end of the URL that you're supplying to the "Datasocket Open (Read)".  In your case: http://finance.yahoo.com/d/quotes.csv?s=BBD-A.TO&d​=t&f=sl1d1t1c1ohgvj1pp2wern[text]
    Also, Jeremy L posted a GetHTMLSourceString.vi.
    Cheers.
    Message Edited by tbd on 01-07-2007 12:08 AM
    "Inside every large program is a small program struggling to get out." (attributed to Tony Hoare)

  • Error when retrieving data from remote url

    Hello,
    I'm trying to do something that should be very simple --
    letting a user enter something in a textbox, then using that data
    as a script parameter when calling a remote cgi script, and
    returning the results to the screen.
    The remote cgi is sending data in plain text. I have looked
    at several examples, and I've tried using an XMLHttpRequest, I've
    tried using the servicemonitor.swf methods, and I've tried with
    URLRequest and URLLoader methods. All fail in one way or another.
    Here's the code for the URLRequest/URLLoader method...
    function fetchISSN(myform) {
    air.trace("Starting Fetch");
    var mytitle = myform.title.value;
    air.trace("TITLE: " + mytitle);
    var reqURL = "
    http://grlinker.coalliance.org/grbin/ilu_rmt.cgi";
    air.trace("URL: " + reqURL);
    var variables = new air.URLVariables("title=" + mytitle +
    "&isf=plain");
    air.trace("URLPARMS: " + variables);
    // var element = document.getElementById('results');
    var request = new air.URLRequest(reqURL);
    air.trace("Request Object Created");
    request.data = variables;
    air.trace("Data Added to request");
    request.method = air.URLRequestMethod.POST;
    var loader = new air.URLLoader();
    air.trace("Loader Object Created");
    loader.addEventListener(air.Event.COMPLETE, loadComplete);
    loader.addEventListener(air.Event.OPEN, loadOpen);
    try {
    loader.load(request);
    } catch(e) {
    alert('Could NOT load data: ' + e);
    } // end of function fetchISSN
    function loadComplete(event) {
    // alert('Downloaded ' + loader.bytesTotal);
    alert('Complete');
    // element.innerText = loader.data;
    } // end of function loadComplete()
    function loadOpen(event) {
    alert('Downloading');
    } // end of function loadComplete()
    And the HTML that calls it...
    <table width="280" border="1" align="center">
    <tr><td align="center">
    <font size="2">Enter a title and click
    submit.</font>
    <form name="issn">
    <input type="text" name="title" /><br>
    <input type="submit" name="submit" value="Submit"
    onClick="javascript:fetchISSN(this.form)" />
    </form>
    </td></tr>
    </table>
    Things go fine until it hits the loadComplete function. At
    that point, the following error is generated...
    Error: The application attempted to reference a JavaScript
    object in an HTML page that is no longer loaded.
    at __HTMLScriptFunction/throwJSObjectDead()
    at flash.events::EventDispatcher/dispatchEventFunction()
    at flash.events::EventDispatcher/dispatchEvent()
    at flash.net::URLLoader/onComplete()
    Can anyone give me any advice or even a reason for this
    failure. I did find a bug about this in the bugtracker, it says it
    is resolved with an "external" resolution. I'm not sure what that
    means. Is there a patch I'm missing? Am I doing something really
    stupid?
    Just for reference, I'm on a Mac running OS X 10.5.6, using
    the SDK for development.
    Thanks!
    Scott

    Hi,
    Can you try putting the declaration of the URLLoader instance
    outside your fetchISSN function and see?
    That is:
    var loader;
    function fetchISSN(myform) {
    //your code above this line
    loader = new URLLoader();
    //rest of your code

  • On my SmarterMail email system, every time I hit "reply" to draft a response to a message, Firefox has begun displaying a mass of HTML data from the original message, making it impossible to use, rather than just the simple text as previously.

    I've been using this web-based email system for many years. Up until last week, every time I wanted to reply to a message, I simply clicked reply, and got a new message draft, with all the previous exchanges neatly tacked on the bottom in reverse chronological order. Now what happens is that I get a mass of HTML gobbledygook, which makes it very difficult to make out what the previous message said. This only affects the "view"ing of the new draft as when I send it, it comes out clean to the recipient. And it only affects my main desktop computer. (Windows XP).

    Hi,
    I have my Contact Testers page which has this HTML
    (opening tag here) a href="aim:goim?screenname=ralphjohnsr&message= Hi+!,+I+came+from+your+website,+ralphjohnsuk." (closing tag here) (opening tag here) img align="right" alt="Webmaster iChat Online Status Indicator" border="0" src="http://big.oscar.aol.com/ralphjohnsr?on_url=http://www.ralphjohns.co.uk//images/ RalphOn.gif& off_url=http://www.ralphjohns.co.uk//images/RalphOff.gif" /(closing tag here)(oen tag)/a(close tag) 
    I have added some linebreaks to make it fit the page here.
    This is obviously done using a valid AIM login name and the a href ="aim:goim? will launch an AIM Client and Message me.
    As you say you would create a New Apple ID then this would be iCloud and an AIM valid Name (keeping the Password to 16 characters or less).
    I am not aware of an iMessage equivalent that would do this to a Phone or iPad or to the Messages Beta as an iMessage.
    To be clear.
    What this HTML does is launch an AIM client (App) on the computer or the person viewing my page and then start a chat with the IM included in the HTML.
    To do that it queries the AIM servers as to whether I am On line or Line and show a pic relevant to that status.  (it does not stop Off Line IMs but I am set up for that.  It also does not say if I am Available or Away.)
    Edit to get html to show (not sure how to work this Raw code thing which dispalyed in the Reply wysiwyg screen.
    8:29 PM      Saturday; March 31, 2012
    Please, if posting Logs, do not post any Log info after the line "Binary Images for iChat"
      iMac 2.5Ghz 5i 2011 (Lion 10.7.3)
     G4/1GhzDual MDD (Leopard 10.5.8)
     MacBookPro 2Gb (Snow Leopard 10.6.8)
     Mac OS X (10.6.8),
    "Limit the Logs to the Bits above Binary Images."  No, Seriously
    Message was edited by: Ralph Johns (UK)
    Message was edited by: Ralph Johns (UK)

  • Iphone died, need to retreive contacts data from itunes ? any advice welcome.

    hello, so the story goes that my iphone just stopped working last weekend. my contacts are in my itunes i hope, as it often sync'd.
    so how do i retrieve my contacts into a format that i could use on another phone ? i currently have an android as a replacement.
    thanks

    Contacts are not stored in iTunes. Contacts are synced through iTunes to a supported contact application on your computer. You have been syncing contacts(as designed) with a supported contact application, haven't you? If not, (bad idea) you'll have to use third-party software to extract your contacts from your iPhone backup. Apple makes no provision to do so:
    http://www.iphonebackupextractor.com/

Maybe you are looking for