Screen scraping

Hi,
How do I 'screen scrape' in java? I don't want to extract information from a web page, but another application, it doesn't have an API and anyway, screenscraping has really caught my imagination! Any pointers, or links would be awesome or in fact, can it be done?
Many thanks, Ron

I did that and to be honest it wasn't really what I was after. The phrase 'screen scrape' is applied very heavily to either interfacing with legacy systems or reading information retrieved from web pages. What I am actually trying to do is observe and transport data from an application running on my desktop to my java application, something which in no close form is covered!
Many thanks for your reply though, Ron

Similar Messages

Screen Scraping in Java

Anybody help me on java screen scrapping.

Svk wrote:
I am asking the best way to screen Scrapping in java !!If you want to scrap your screen, take it to an electronics recycling.
If you want to do screen scraping, you could provide some more detail about what you actually are trying to accomplish. Note: exclamation points aren't details.
I'll bet what you want to do isn't even screen scraping. It's becoming increasingly rare these days.

RE: (forte-users) Screen Scraping

I think they changed their name since Conextions at least 2 times. Try this
web site:
http://www.wrq.com/products/apptrieve/
noro
-----Original Message-----
From: Denver Jobs [mailto:fortejobsindenveryahoo.com]
Sent: Monday, November 13, 2000 8:31 AM
To: forte-userslists.xpedior.com
Subject: Re: (forte-users) Screen Scraping
Michael,
We use Conextions for screen scraping from the
mainframe - and I believe that it is supported by
Forte. I tried checking their website to see if they
offered it for Unix, but their site was down.?
Some others you may want to check with are Pixel
Science Technology or FileFrameVX. I also saw an
interesting article on Eclipse Host Integration
Server, but it appears they've shut down and
transferred their knowledge base to a company called
eNucleus.
--- Michael Strauss <mstraussmazda.com.au> wrote:
Has anyone performed any screen scraping either
directly from Forte
(sockets) or via C wrapping? I would be very
interested to hear some
feedback and pointers. I want to scrape from my
Unix server and provide the
scraped data to my GUI client (Win).
TIA
Michael Strauss
>
>
Mazda Australia takes every precaution to ensure
email messages
are virus free.
For extra protection you should virus scan this
message yourself.
>
For the archives, go to:
http://lists.xpedior.com/forte-users and use
the login: forte and the password: archive. To
unsubscribe, send in a new
email the word: 'Unsubscribe' to:
forte-users-requestlists.xpedior.com
http://calendar.yahoo.com/
For the archives, go to: http://lists.xpedior.com/forte-users and use
the login: forte and the password: archive. To unsubscribe, send in a new
email the word: 'Unsubscribe' to: forte-users-requestlists.xpedior.com

Hi Jean-Paul,
As described in the Technote 10981 some Forte programs (Nodemanager and
router) handle correct the high-file descriptor-use problem. It is possible
that Forte interpreter do it correct too.
Zenon
-----Original Message-----
From: Jean-Paul Gabrielli [SMTP:Jean-Paul.Gabriellisema.fr]
Sent: Monday, September 25, 2000 12:11 PM
To: Adamek, Zenon
Cc: Forte-userslists.xpedior.com
Subject: RE: (forte-users) [UNIX] "Too many open files" 3.0.M2
question
Actually, the stuff works in interpreted mode.
It's only when having the server partition compiled that this happen.
j-p
-----Message d'origine-----
De: Adamek, Zenon [mailto:ZAdamekpurolator.com]
Date: lundi 25 septembre 2000 17:13
À: 'Jean-Paul.Gabriellisema.fr'
Cc: Forte-userslists.xpedior.com
Objet: RE: (forte-users) [UNIX] "Too many open files" 3.0.M2 question
see Technote 10981
-----Original Message-----
From: Jean-Paul Gabrielli [SMTP:Jean-Paul.Gabriellisema.fr]
Sent: Monday, September 25, 2000 11:02 AM
To: zeForte-users
Subject: (forte-users) [UNIX] "Too many open files" 3.0.M2 question
Hi,
running a server partition that reads a configuration file,
and apparently doen't close it after, I have that exception:
SYSTEM ERROR: System Error: Too many open files, opening '....'with mode
'r'
Class: qqos_FileResourceException
1) Is there such a limit, or does this rely only on the OS one ?
2) How is this error not trapped, as I only got itinteractively, whereas
my server log does a exception trap/segmentation fault,
thanlks
j-p
For the archives, go to: http://lists.xpedior.com/forte-users and use
the login: forte and the password: archive. To unsubscribe,send in a new
email the word: 'Unsubscribe' to:
forte-users-requestlists.xpedior.com
>
For the archives, go to: http://lists.xpedior.com/forte-users and use
the login: forte and the password: archive. To unsubscribe, send in a new
email the word: 'Unsubscribe' to: forte-users-requestlists.xpedior.com

Consumption of data model created using Screen Scraping

Dear All,
I have created a GW Data Model for recording a standard SAP transaction and subsequently created its GW Consumption Model in SAP Netweaver Gateway.This will return a REST service.This REST service supports 'CRUD'
(CREATE,READ,UPDATE.DELETE) operations as per the documentation .While using this REST service I am facing two
problems.
Case 1:
I created a GW Comsumption Model by recording transaction 'TZ60' for READ operation,created QUERY operation
and 'TZ90' for UPDATE operation. I can use the READ operation successfully in Dot Net Application but how can I use
the UPDATE operation in Dot Net Application ?
Case 2:
Can wev create Screen Scraping Data Model for complex transactions involving Multiple Screens/Tabstrips and ALVs.
Please reply to this thread as soon as possible.Thankyou.

hi Chris
Before you might want to start refactoring a copy of what you already have, you could consider this ...
"Fear for renaming (refactoring) ADF BC objects in JDeveloper 10.1.3.3 is not unfounded by Frank Houweling"
http://technology.amis.nl/blog/?p=2859
success
Jan Vervecken

Can OIM do Screen Scraping?

Does OIM has Screen Scraping ability ? Or how do we achieve the Screen scraping using OIM ?
Screen scraping will be dynamic in nature, in the sense that we will have different screens for different users, based on their entitlements.
The use case is to read the screen first and analyze, then feed the screens based on the provisioning request.
Any suggestions on this can be extremely useful to us.
Thanks.
Edited by: user5512723 on Mar 18, 2010 12:21 AM

If I understand ur question correctly then u want to show different pages to different users based on groups/roles etc.
But this can't be possible w/o UI Customization. You have to lots of UI Customization or your custom web application.
In OIM you can give different menu items based o their group etc to perform different operations.

Screen scraping mainframe terminal emulator/web page

Hi,
Is it possible to screen scrap the contents of mainframe Terminal emulator.
I have a 3270 Terminal emulator. What is the best way to read the content of this terminal emulator.
I'm aware that we can screen scrape static or dynamic web pages. But, is it possible to screen a
terminal emulator. I have no clue reg this. Plz help.

Thanks for your response DrClap!!
But you would have to be working with a published interface, as I was. Does >>your 3270 emulator have a published interface that allows access to the >>screen display?We are using Bluezone terminal emulator (from seagull software). This Bluezone uses an ActiveX control or Java applet to download a Win32 emulation application.
Given this scenario, plz let me know how do I screen scrap the contents of this terminal emulator and pass the screen scraped content to another java program.
Plz help so that I go in the right direction.
(For screen scraping web pages: I was using JTidy and XQuery. But, for this terminal emulator, how do i read the request url. I'm a beginner, so excuse me if I'm completely wrong with this approach).
Thanks,
Sid

Screen scraping and IPC

Hi there,
We have a mainframe emulator which is implemented as a webapp. This webapp gets some data from mainframe say reservation #.
I tried creating a browser portlet for the webapp (mainframe app).
Now, there are 4 other JPF portlets which listen to the above webapp or browser portlet for reservation # . I' guess there is no way to share the data between browser portlet and other portlets. (Plz correct me if I'm wrong).
My question is:
1. Is there any way I can pass the data retrieved from external webapp into my portal framework i.e to other portlets
2. Will Screen scraping or streaming web page content help in this scenario.
Plz share your thoughts. Appreciate any help.
Thanks,
Sid

Sid,
It is possible, but you need to 'suck' the data from the mainframe app
into a JSP-portlet. You can do this in the backing context of the
JSP-portlet.
import javax.servlet.http.HttpSession;
import javax.servlet.http.HttpServletRequest;
import org.apache.beehive.netui.pageflow.PageFlowController;
import org.apache.beehive.netui.pageflow.annotations.Jpf;
import org.apache.beehive.netui.pageflow.Forward;
import org.apache.beehive.netui.pageflow.scoping.ScopedServletUtils;
@Jpf.Controller(simpleActions = { @Jpf.SimpleAction(name = "begin", path
= "retrieveParams.do") })
public class PfIpcTestController extends PageFlowController {
     private static final long serialVersionUID = 1961633610L;
     @Jpf.Action(forwards = { @Jpf.Forward(name = "success", path =
"displayParams.jsp") })
     public Forward retrieveParams() {
          HttpServletRequest request =
ScopedServletUtils.getOuterRequest(getRequest());
          String cloneMessage = request.getParameter("_cloneMessage");
          CloneMessage msg = new CloneMessage();
          if( cloneMessage != null )
               System.out.println("Inside retrieveParams: cloneMessage="+cloneMessage);
               msg.setTheMessage(cloneMessage);
          else
               msg.setTheMessage("No clone message received as URL parameter");
          Forward forward = new Forward("success");
               // Adding the message as an output form isn't strictly
               // required but is handy if another application (or this
               // pf) wants to register a handler on this action and
               // get this payload
          forward.addOutputForm(msg);
          forward.addActionOutput("_cloneMessage", msg);
          return forward;
     * Callback that is invoked when this controller instance is created.
     @Override
     protected void onCreate() {
     * Callback that is invoked when this controller instance is destroyed.
     @Override
     protected void onDestroy(HttpSession session) {
     @Jpf.FormBean
     public static class CloneMessage implements java.io.Serializable {
          private static final long serialVersionUID = 838010769L;
          private String theMessage;
          public String getTheMessage() {
               return theMessage;
          public void setTheMessage(String theMessage) {
               this.theMessage = theMessage;
This way you clone the content of the original page, and bring it in the
context of the WLP. Now you can create IPC. The only thing you need to
do is to post the user clicks back to the backing file which the submits
the correct values to the mainframe app.
Best Regards,
Frank van den Nieuwboer
Hi there,
We have a mainframe emulator which is implemented as a webapp. This webapp gets some data from mainframe say reservation #.
I tried creating a browser portlet for the webapp (mainframe app).
Now, there are 4 other JPF portlets which listen to the above webapp or browser portlet for reservation # . I' guess there is no way to share the data between browser portlet and other portlets. (Plz correct me if I'm wrong).
My question is:
1. Is there any way I can pass the data retrieved from external webapp into my portal framework i.e to other portlets
2. Will Screen scraping or streaming web page content help in this scenario.
Plz share your thoughts. Appreciate any help.
Thanks,
Sid

Screen Scraping embedded Siebel ActiveX Controls...

I provide Tech Support for Siebel 7.8 for our Sales Force. My manager has asked me to look into a way to "Screen Scrape" data from specific fields. Since the data is presented in activex controls I don't believe a typical screen scrape is possible with all of the frames and controls I'd have to drill through to get to what I want. Is it possible to use something like the FindWindow api or something similar to retrieve the values I need?

Ok I have a more specific question, I've been able to use DOMExplorer to find the specific activex control I want to retrieve data from. Is it possible to extract data from the control by using it's IDispatch interface or some other interface? I'm reading up on the architecture of ActiveX controls but I haven't read anything that explicitly says whether this is possible.
Caeanis

Screen scraping in Portal

I have a requirement to screenscrape various different HTML and ASP sites and display
them as a portlet. What is the best way to do this?
I tried frames, but then I lose the header and navigation. I tried wl:include
and that does not seem to work.
Thanks in advance
MG

Hi Mike,
Please take a look at Kapow:
http://www.bea.com/framework.jsp?CNT=index.htm&FP=/content/products/third/kapow/
Regards,
--alex
"Mike Greigo" <[email protected]> wrote:
>
I have a requirement to screenscrape various different HTML and ASP sites
and display
them as a portlet. What is the best way to do this?
I tried frames, but then I lose the header and navigation. I tried wl:include
and that does not seem to work.
Thanks in advance
MG

Web Data Scraping with Adobe Flex?

Hi all.
I need to get a lot of data coming from several websites.
For now, I'm using Djuggler, but I'm wondering if something simular can be achieved with Adobe Flex?
I have no experience with Adobe Flex in this field, so any remarks, comments, suggestions, tutorials, would be much much appreciated!
I need to have good arguments to convince my boss to buy Adobe Flex :-)
Thanks a lot in advance!!!
Kind Regards
Jurgen Landrie

No. Read-access won't help. What is required is that there is a webservice available for your use.
For example, you can go to Yahoo and view stock quotes via the Yahoo finance page. And yes, you can screen-scrape it, but Yahoo also provides a webservice to access the quotes directly. Flex is designed to make calls to those webservices and display the data. IIRC, the terms and conditions of many websites discourage or disallow screen-scraping.

Screen Damage Due to "Blackberry" Badge on Closure Flap of 8820 Swivel Holster

I recently received a Blackberry 8820 which came with the lambskin swivel holster in the box. The closure flap of this holster has a metal badge on it with the "Blackberry" name cut into it. The surface of this badge, especially where the name is cut, is very rough.
The other day, while placing the phone back in the holster, the closure flap got inadvertently tucked inside the case and the surface of the screen scraped across the metal badge. I could hear and feel this as it happened and stopped inserting the phone immediately, but now there is a huge gouge in the screen. At first I was sick about it; now I am angry because the damage was caused by the case that was intended to protect it.
I contacted my carrier (T-Mobile) who gave me the "too bad; so sad... there's nothing we can do" treatment
I feel this was the result of a poor design and would, at a minimum, like to contact RIM but cannot seem to find a direct number.
1) Has anyone else experienced this?
2) Is there any recourse?
3) Does anyone know how I can contact RIM directly to suggest they move this badge or change the design to prevent this from happening again.
Thanks in Advance

Most of my users have their phones with the screen facing out.
This way if the tab gets caught, it would only scratch the back. Plus then they can see the notification light.
Suppose you could try contacting RIM... 519/888-7465.
Is your phone under still under contact? I've had Verizon replace 8830's that were busted for $50.00.
The other option is you can buy the front piece plastic piece that covers the screen, called a Front LCD Screen and replace it. You can pick them up for about $6.49 on Ebay. Then go to Youtube to find a video on how to install it.

Screen scrape Oracle 9iAS web based forms

Hello Friends,
We are currently in the coding stage of a screen scraping application.But we are facing a serious problem,that might cause us to abandon this project altogather.We are supposed to screen scrape Oracle 9iAS based Oracle Forms,precisely speaking we have to trace the Msgboxes and Statusbar warnings,which are thrown at runtime during any transaction by the system.Could anyone help?
Please reply ASAP.
Thanks and Regards,
Samir P
Bangalore

Hi,
though this is not security related: You can use JavaBeans within a Bean Container in Oracle Forms. This allows you to add any Java you like to the Forms front end. However, you may have to dig a bit into the Forms client for this.
Frank

Query to extract HTML tag with data

Hi All,
I have a string.
'<HTML><HEAD>THIS IS HEAD.</HEAD><BODY>THIS IS BODY.THIS IS P1.NIMISHTHIS IS P2.</BODY></HTML>'
I want to extract a html tag including its opening & closing tab with data as
if i say P1
then the output should be
'THIS IS P1.'
for P2
then the output should be
THIS IS P2.
please help me in writing this query with regular expression
i have tried it as following but it is not giving desired result:
WITH T AS
SELECT
 '<HTML><HEAD>THIS IS HEAD.</HEAD><BODY>THIS IS BODY.THIS IS P1.NIMISHTHIS IS P2.</BODY></HTML>' STR
FROM
 DUAL
SELECT REGEXP_SUBSTR(STR, '.+P2.+') FROM T
Thanks & Regards
Nimish GargEdited by: Nimish Garg on May 7, 2012 5:49 PM

Nimish Garg wrote:
My requirement is to extract a <tag>data</tag> from a HTML/XML string
where data contains any specified value.HTML is not XML.
And that is a critical distinction to make. HTML parsing is horribly complex. XML is quite easy. For HTML you have to code your own parser in PL/SQL. XML can be parsed using the XMLTYPE class/data type in PL/SQL.
So if you need to find a single specific tag in HTML - I would not try to treat it as XML. I may not even try to use regular expressions.
I would do a basic substring search for the start of the tag. Read the data following the tag. Ensure that there are no nested or embedded tags in the data. Until the end tag is read. Because HTML is that much abused - and because that is an accepted norm as parsers used by browsers deals with that abuse without complaining.
Proper HTML is mostly a myth in my experience of "screen scraping" web servers for data extraction as they do not have web services supplying the data.

How to create a global role with WLST in WL 10

Hi All:
The approach in the protect_resources.py found in dev2dev, doesn't work. I've managed to convert the user and group creation to work with WL 10, but I can't for the life of me figure out how to create the role. There doesn't seem to be a createRole() on what I would think are the appropriate MBeans in poking around.
Anyone know how to do this, or will I have to come up with a screen scraping solution that does this via the weblogic console, where it's so easy to do so?
TIA
Forrest

Not having X's programing background I think of an action reference as something that tells Photoshop what to do. And yes it is like a little action that you write instead of record. For example the code that I and X posted could also be written like this
var ref = new ActionReference();
ref.putProperty( charIDToTypeID( "Prpr" ), stringIDToTypeID('tool') );// what key to get
ref.putEnumerated( charIDToTypeID("capp"), charIDToTypeID("Ordn"), charIDToTypeID("Trgt") );// where to get it from
var cTool = executeActionGet(ref);// in this case returns a one key descriptor
var cToolTypeID = cTool.getEnumerationType( stringIDToTypeID('tool') );// get the value of that key
alert( typeIDToStringID( cToolTypeID ) );// make that value readable
Most of the ordinals you will see will be target as Photoshop likes whatever you are working on to be active. You sometimes see next or previous. I can't recall seeing a 'normal ordinal' like first or second.
There is not much in the way of documentation. Most of what I know comes from looking at the scriptlistner log, xtools and X himself, and a little bit of code I put together for exploring action descriptors and action list. It's not as nice as X's getterdemo but works more the way I think. It sends it's output to the ESTK console window
var ref = new ActionReference();
ref.putEnumerated( charIDToTypeID("Lyr "), charIDToTypeID("Ordn"), charIDToTypeID("Trgt") );
var desc = executeActionGet(ref)
var c = desc.count ;
//for(var i=0;i<c;i++){ // to enumerate list
// $.writeln('Key '+i+' = '+desc.getType(i))
for(var i=0;i<c;i++){ //enumerate descriptor's keys
$.writeln('Key '+i+' = '+typeIDToStringID(desc.getKey(i))+': '+desc.getType(desc.getKey(i)))

How can I extract information from an IdM approval stack?

During the workflow for provisioning employees, managers are required to authorize certain actions. The problem is that they sometimes do not.
My boss has asked for a formatted report in ascending date order that shows the passive authorizations (some are weeks old!) and the managers to whom the "lazy" managers report.
I've done this, but it's driven by my manually screen-scraping the information from the approval list onto an external file.
The problem is that I have to initiate the process by bringing up the approval screen, copying and pasting information into a file and then running a Perl munge script against the created file...
I want to be able to acccess the approvals list by means of a crontab initiated script and not have to rely on the "screen-scrape."
Any ideas?
Ian Clark
ADP - ProBusiness.
Message was edited by:
Namalankula

You can do it in following way....
First you need to create the object of URL by specifying the URL name.
URL myurl = new URL("http://www.xyz.com/index.html"); //specify your URL path here
URLConnection conn = null;
DataInputStream data = null;
String line;
StringBuffer buf = new StringBuffer();
try {
conn = myurl.openConnection();
conn.connect();
data = new DataInputStream(new BufferedInputStream(
conn.getInputStream()));
while ((line = data.readLine()) != null) {
buf.append(line + "\n");
data.close();
catch (IOException e) {
System.out.println("IO Error:" + e.getMessage());
So, at the end, you have the data in your string buffer...you can use it wherever u want.
Hope this helps

Screen scraping

Similar Messages

Maybe you are looking for