Authenticating a crawler thru a http password request prompt

I have a crawler that I want to authenticate thru a http password request. To explain the situation a little more, I have several links to various reports on a server that requires me to authenticate thru a http password request. (When I click the links directly a prompt appears asking me for creditials. Not a web form!). Previously, I set up crawlers to go thru web form authentication with success. Is it possible to pass my creditial from a web crawler to the http password prompt? If more clarification is needed, just let me know. Thanks in advance.
Terrel

what he is trying to say is that if you want help with your project, change it. You will not get any help here to write code that can be used to do things like spam, be used as a bot, phishing or any other illegal/invasive activity.
Any 'normal' project should have no business logging into a website programmatically; the login prompt is there for a reason, to provide security. What you are trying to do is break through it. Can you see how this will make people frown?

Similar Messages

Using ADFS authentication to perform SSO via HTTP GET request

Hi,
Can i authenticate users (those users are clients, at home) to a web application using ADFS without SAML tokens?
The situation is that i want the clients to perform SSO to the website via a link they receive in their mailboxes.
I thought about a solution that combines JWT in a URL link that each user will get to his private mail. this link will contain the users' claim (such as ID Num, given from AD DS Server dedicated especially for them).
Thus, the user will receive an email with a link that already contains a short period of time JWT to perform SSO to the webapp.
Is it possible ? anybody heard about a similar solution ?

Sandra
Thanks for your message
Here is the my requirment
The basic flow of a Where 2 Get It REST API call is:
1) create the required XML structure,
2) URI encode it,
3) make a HTTP GET request,
4) then parse the return XML document.
Currently i have some data in ABAP structure with 5 fields, i need to create XML from the those 5 fields,and needs to be URI
encode it, and then needs to make a HTTP get request to connect Where to Get It REST API, finally it will return XML document via HTTP Get request , and then needs to convert the return XML to ABAP structure for further processing .the above 4 points will be implemented in my report.
Any body could help on this

HTTP PUT Request for File Uploads is returning a response header with content-length = 0

After upgrading from CF9 to Coldfusion 10, I'm running into an odd issue, with `cURL` calls to a webservice that used to work.
We are using an HTTP PUT Request with multipart="no" so that the request header content is the file itself, now the data is being sent properly in the content but it has a ~8k limit. This is fine for smaller files but some files are 60kb etc.
Errors are odd too, near the 8KB limit we get a bad gateway error 502, and above that the webpage errors with a connection reset.
It works fine with multipart="yes", but in a PUT request the file's data isn't even sent since it's looking for `httpparam formField` , the content-length from the target page's Headers show a content-length of 0 which causes `cURL` to fail with an error (56): http://curl.haxx.se/docs/manpage.html
Here's example code to reproduce:
<cfset sFilename = "C:\999\test4.txt">
<cffile action="readBinary" file="#sFilename#" variable="fileRead">
<cfset oFileInfo = GetFileInfo("#sFilename#")>
<cfhttp           url="localhost/Clarence/diff_elt_test_output.cfm"
                              method="PUT"
                              username="******"
                              password="******"
                              multipart="yes"
                              result="oHttp">
          <cfhttpparam type="header" name="Content-Type" value="text/plain" />
          <cfhttpparam type="header" name="Content-Length" value="#oFileInfo.Size#" />
          <cfhttpparam type="body" value="#fileRead#"/>
</cfhttp>
<cfdump var="#oHttp#">
Dumping the response via GetHttpRequestData() shows that the header's content-length = 0 and content is blank.
Whereas POST requests work fine, cURL has a -F form option which is our fallback but a solution to this would be great.
Here's what I've tried:
1. Changing CF post request limit sizes (64MB+)
2. Changing IIS config web/webserver http runtime and security.request filtering request limits to much higher values
3. AJP Connector between IIS and Tomcat has a default 8KB header limit, I changed that to 64k on both sides, still nothing
Also am I missing something fundamental here? I thought at the network layer a packet is split up and rejoined, so large content in a http request body should work, what's wrong with Coldfusion 10?

I am familiar with java.net package and know how to
Read and Write Streams.
However, what I do not know is what string
represntation of PUT or POST resquests is?It's HttpURLConnection.setRequestMethod("PUT") or ("POST").
Also you have to setDoOutput(true).
Then you set the content type, possibly encode the input, set the content length, and write the file data to the stream.
Using a Jakarta package may be easier, however.

HTTP POST Request with XML file in

Hi @ all,
I would like to send an HTTP Request with an XML File in the body to an SAP System
I have the target URL and the a XML File.
Now the question is. Is it possible to use something like the HTTP_POST FuBa to send an url post request with an xml file?
If yes can anybody give me a hint?
I have a php script which exactly do this coding. But to integrate it all in one system it is necessary to transform it into ABAP and call it there.
// compose url and request and call send function
function test($productID, $categoryID) {
     // create url
     $PIhost = "XXX.wdf.sap.corp";
     $PIport = "50080";
     $PIurl = "/sap/xi/adapter_plain";
     $PIurl .= "?sap-client=800";
     $PIurl .= "&service=XXX";
     $PIurl .= "&namespace=XXX";
     $PIurl .= "&interface=Frontend_Interface";
     $PIurl .= "&qos=EO";
     $PIurl .= "&sap-user=XXX";
     $PIurl .= "&sap-password=XXX";
     // create xml
     $request = "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n";
     $request .="<n1:FrontendInbound_MessageType xmlns:n1=\"http://crmpiebay.com\">\n";
     $request .= "\t<FrontendInbound>\n";
     $request .= "\t\t<ProductName/>\n";
     $request .= "\t\t<ProductCategory>".$categoryID."</ProductCategory>\n";
     $request .= "\t\t<ProductID>".$productID."</ProductID>\n";
     $request .= "\t\t<MessageID/>\n";
     $request .= "\t</FrontendInbound>\n";
     $request .= "</n1:FrontendInbound_MessageType>";
     // send http request
     postToHost($PIhost, $PIport, $PIurl, $_SERVER["SERVER_NAME"].":".$_SERVER["SERVER_PORT"]."/".$_SERVER["PHP_SELF"], $request);
// send post request to PI server
function postToHost($host, $port, $path, $referer, $data_to_send) {
$fp = fsockopen($host, $port);
if($fp) {
     $res="";
       fputs($fp, "POST $path HTTP/1.1\r\n");
       fputs($fp, "Host: $host\r\n");
       fputs($fp, "Referer: $referer\r\n");
       fputs($fp, "Content-type: text/plain\r\n");
       fputs($fp, "Content-length: ". strlen($data_to_send) ."\r\n");
       fputs($fp, "Connection: close\r\n\r\n");
       fputs($fp, $data_to_send);
       while(!feof($fp)) {
           $res .= fgets($fp, 128);
       fclose($fp);
       return $res;
Would be great if anybody could give me a hint how to solve such a HTTP Post request with XML body data.
Thanks in advance.
Chris
Edited by: Christian Kuehne on Feb 26, 2009 4:32 PM

hi friend could you please share your solution regarding this query if u got it already?

How to enable password request for restart and shutdown?

Hi,
I'm neither a Linux nor an Arch Linux newbie, but inexperienced regarding this particular issue.
I want to enable password request for restart and shutdown and want to know if I'm mistaken, because trial and error might become to time-consuming while working on a project.
I read https://wiki.archlinux.org/index.php/Al … o_shutdown.
$ ls -hAl /usr/bin/shutdown
lrwxrwxrwx 1 root root 9 Apr 22 03:02 /usr/bin/shutdown -> systemctl
If I try to run $ shutdown -hP 28 or $ shutdown -c nothings happens, I need to run $ sudo shutdown -hP 28 and $ sudo shutdown -c and type a password. That's the way I want it.
If I e.g. run $ shutdown -r now no password is needed. I want to disable this. It should behave the same way as shutdown -hP/-c behave. I want to type
$ sudo shutdown -r now or $ sudo systemctl reboot etc. and then the password should be required.
$ sudo grep -vn "#" /etc/sudoers | grep [[:blank:]]
72:root ALL=(ALL) ALL
73:rocketmouse ALL=(ALL) ALL
The user "rocketmouse" should have all permissions after typing a password, but not without typing the password.
IIUC what's written at https://wiki.archlinux.org/index.php/Polkit, I need to edit
$ pkaction | grep login
org.freedesktop.accounts.set-login-option
org.freedesktop.login1.attach-device
org.freedesktop.login1.flush-devices
org.freedesktop.login1.hibernate
org.freedesktop.login1.hibernate-ignore-inhibit
org.freedesktop.login1.hibernate-multiple-sessions
org.freedesktop.login1.inhibit-block-idle
org.freedesktop.login1.inhibit-block-shutdown
org.freedesktop.login1.inhibit-block-sleep
org.freedesktop.login1.inhibit-delay-shutdown
org.freedesktop.login1.inhibit-delay-sleep
org.freedesktop.login1.inhibit-handle-hibernate-key
org.freedesktop.login1.inhibit-handle-lid-switch
org.freedesktop.login1.inhibit-handle-power-key
org.freedesktop.login1.inhibit-handle-suspend-key
org.freedesktop.login1.power-off
org.freedesktop.login1.power-off-ignore-inhibit
org.freedesktop.login1.power-off-multiple-sessions
org.freedesktop.login1.reboot
org.freedesktop.login1.reboot-ignore-inhibit
org.freedesktop.login1.reboot-multiple-sessions
org.freedesktop.login1.set-user-linger
org.freedesktop.login1.suspend
org.freedesktop.login1.suspend-ignore-inhibit
org.freedesktop.login1.suspend-multiple-sessions
org.freedesktop.machine1.login
IOW I need to replace every yes and no etc. with auth_admin in $ grep -v lang /usr/share/polkit-1/actions/org.freedesktop.login1.policy.
Am I mistaken?
Regards,
Ralf

You'll need to create a rules file which uses javascript.
https://wiki.archlinux.org/index.php/Po … tion_rules
// /etc/polkit-1/rules.d/10-admin-shutdown-reboot.rules
polkit.addRule(function(action, subject) {
if (action.id == "org.freedesktop.login1.power-off" ||
action.id == "org.freedesktop.login1.power-off-ignore-inhibit" ||
/*...SOME_MORE_IDS_HERE...*/
// return polkit.Result.AUTH_ADMIN_KEEP;
return polkit.Result.AUTH_SELF_KEEP;
Last edited by progandy (2015-06-21 17:42:35)

Block HTTP Options request in DBMS_EPG

We are having some trouble opening HTML pages from Office tools in combination with the Embedded PL/SQL Gateway on a Oracle 11g database.
When we open a public Apex page from word or Excel it will prompt for XDB username / password.
We open the page like this: http://epg-host:8080/apex/f?p=111:1
When we open the same page from the same Apex application using the Oracle HTTP Server instead of the Embedded PL/SQL Gateway, Word and Excel won't prompt for a username and password.
I used Wireshark to see the difference in HTTP traffic. Microsoft Office will do a HTTP Options request on the "directory" of the page (for the url http://epg-host:8080/apex) before opening the page. The HTTP Options request results Error 401 Unauthorized, that’s why Word/Excel ask the user to login.
With a HTTP Send Tool I have send the Options Request to the Embedded PL/SQL Gateway and to the Oracle HTTP Server, these are the different responses:
Options for http://epg-host:8080/apex/
The remote server returned an error: (401) Unauthorized.
MS-Author-Via     DAV
DAV     1,2,<http://www.oracle.com/xdb/webdav/props>
Content-Length     147
Content-Type     text/html; charset=UTF-8
Date     Thu, 10 Nov 2011 10:05:56 GMT
Server     Oracle XML DB/Oracle Database
WWW-Authenticate     Basic realm="XDB"
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<HTML><HEAD>
<TITLE>401 Unauthorized</TITLE>
</HEAD><BODY><H1>Unauthorized</H1>
</BODY></HTML>Options for http://apache-host:7778/apex/
The remote server returned an error: (501) Not Implemented.
Allow
Connection     close
Content-Length     252
Content-Type     text/html; charset=iso-8859-1
Date     Thu, 10 Nov 2011 10:15:44 GMT
Server     Oracle-Application-Server-11g
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>501 Method Not Implemented</title>
</head><body>
<h1>Method Not Implemented</h1>
<p>The server does not support the functionality required to fulfil the request.</p>
</body></html>I do not really understand why the Options Request results in Unauthorized. I would like to change the behavior of the Embedded PL/SQL Gateway so that it does not throw the "Unauthorized" error. Any other error would be good, but Unauthorized results in a Login prompt.
Does anyone know how to configure the Embedded PL/SQL Gateway so it will not result in the unauthorized error? It doesn’t matter if I have to block the Options requests or grant extra authorization.

Thanks for your reply. When we open the page in the browser we do not get the Login-prompt, the allow-repository-anonymous-access is already set to true.
The login-prompt only occurs when we open the page from Word/Excel, that's when the HTTP-Options request is send.
Opening an Apex page in the browser only results in a HTTP-Get request, Word/Excel will send HTTP-Options followed by the HTTP-Get request.

How to automize Self-control [Internet blocker] / bypassing password request?

Hey there,
I hope someone on here may be able to help with this. I figured it might be good to ask about this on a popular board geared towards general non-technical user like myself.
I use the app SelfControl [http://selfcontrolapp.com] to help me manage my internet addiction. The app's great. It really helps--when it's running.
I've been trying to set it up so it automatically starts at certain times of the day [i.e. every day 10pm-7am], Using the guide found here [ http://hints.macworld.com/article.php?story=20100801214648362,] I've been able to integrate SelfControl into my iCal, to set it up as a recurring iCal event.
But SelfControl requires my admin password to actually initiate the the internet block. So right now, I have to manually enter the password every time iI want to run SelfControl--even when it's automatically launched as an event from iCal. This kind of makes the app useless as I can just not enter the password and continue surfing endlessly.
Basically, I need some help figuring how to bypass the password request/have it inputted automatically. I did try following the instructions found in the above link about using keychain access, etc but none of it seemed to work--SelfControl still prompts me for my password.
I am hoping someone here can help me out; the instructions in the above page are a little too technical for me to fully comprehend (and the forum linked above is now permanently closed)
I'm the only user of my laptop, if that makes a difference. I am running OS 10.10.1 Yosemite.
Any help would be appreciated
Thanks,
Kursty

so many views and so few answers... Okay, most of those were me refreshing the page, but still..... HELP

Imac goes back to password request screen

imac goes back to password request screen over and over again after an upgrade notification " CS5 Service Manager needs Java 6 " . This happened yesterday after running a software upgrade, somehow after so many tries and repairing disk, using RP, etc it started to work. TODAY, another software upgrade with a "malware removal' file came and I thought Apple had found and solved the problem....WRONG I went to the same stae again....going crazy....please help. Thank you

Hi Mauricio,
I remember a sim. problem on MacOS 10.6. I remember that by then it was Firefox which needed to be upgraded to use the systems JRE (Java Resource Engine) instead of some other floating around on the OS. Maybe the conclusions drawn in the thread (external link) can help you, as this sounds a little like your problem: http://forums.macrumors.com/showthread.php?t=1153590
Furthermore, I urgently advice you to scan your computer for viruses and malware programs. MacScan is offering a trial mode, where you can do this for free. Hope that helped.

HTTP web request error (Web exception for remote server)

To whom it may concern,
I have recently applied a few patches for my client's server for IE9 but then one application starts to malfunction right after upgrade return successful. The program affected is CyberTech player which make use of Microsoft Silverlight.
My client currently uses Wind 2008 R2 x64, patches applied are below:
MS13-009-IE9-Windows6.1-KB2792100-x64
MS13-010_IE9-Windows6.1-KB2797052-x64-sp1
MS13-021_IE9-Windows6.1-KB2809289-x64
MS13-028IE9-Windows6.1-KB2817183-x64
MS13-037_IE9-Windows6.1-KB2829530-x64
MS13-038_IE9-Windows6.1-KB2847204-x64
MS13-047_IE9-Windows6.1-KB2838727-x64
MS13-069_IE9-Windows6.1-KB2870699-x64
Please let me now if any further information is needed.
Your prompt response will be much appreciated.

Hi,
Based on your description, I would like to suggest you try to uninstall and reinstall the latest version of Microsoft Silverlight.
http://www.microsoft.com/silverlight/
I also would like to suggest you update the CyberTech player so that it can be compatible with this update.
Meanwhile, the http web request error was caused by IE itself:
1. Improperly install or configure updates
2. Security website authentications issue
3. IE settings
You may try the following steps to check the issue:
1. Run IE with no add-ons. Click Start -> All Programs -> Accessories -> System Tools -> Internet Explorer (with no add-ons).
2. Try to use compatibility mode (go to tools --> Compatibility View Settings) to check the issue.
For further research, it would be kind of you to capture a screenshot of the error page.
Hope it helps.
Regards,
Blair Deng
Blair Deng
TechNet Community Support

Extract Portal User Name From HTTP POST Request (WSRP SOAP)

How can we extract the user name from the HTTP POST Request (*WSRP SOAP*) from the Portal to the Portlet Provider (Producer)?
We are load balancing between the Portal (10.1.14.2) and the providers (OC4J 10.1.3.3) and would like to extract the user name within the load balancer so we can create session affinity based on it.
BTW, we already tried using TCP Flow but were unable to see the user name.

From the documentation, it seems that you need to use
HttpURLConnection.setRequestMethod("POST"); // "GET" is the defaultalthough I have seen examples on the web without it.
Try!
Edit: Also the single quotes around userid and password may be a problem.
Use the previous poster suggestion, but do not include the single quotes.
Edited by: baftos on Aug 6, 2008 4:27 PM

Sending email in one account, triggers a password request from another

I have several email accounts operating in MAIL. I have recently noticed that when I send an email in one account, a second account pops up and requests a password. I used to type in the password, but then I thought: has a nasty virus infected my system and somehow, whenever I send emails, they are also secretly being sent somewhere else via this second password-seeking account.
There is nothing in the Outbox for the second account, so I'd like to find out what is going on. Is this an indication of a virus, or has MAIL gone haywire?
I can get around the problem by clicking OK to the password request from the second account (but not having typed anything in), which sends me to a window asking me if I want to use another account to send the email. I select the first account (the one I actually want to use), it asks me for the password, and away it goes.
Thanks in advance for any comments.
Guy Burns

Guy,
You need to provide us with some additional info about the SMTP you are using as the Outgoing Server for each account, and what, if any, relationship that SMTP has to both accounts. The use of the SMTP provided by your ISP, for example, if Authenticated either by Password or by SMTP after POP3 methods, can trigger a password prompt that looks the same as that for the Incoming server of an account provided by that ISP.
More info, please.
Ernie

My ipad keeps having the Apple ID Password request box pop up? How do I stop it doing this?

The Apple ID Password request box keeps poping up. I continually have to press cancel to make it disappear? How do I stop my Ipad from doing this?

Try This...
Close All Open Apps... Sign Out of your Account... Perform a Reset...
Reset ( No Data will be Lost )
Press and hold the Sleep/Wake button and the Home button at the same time for at least ten seconds, until the Apple logo appears. Release the Buttons.
http://support.apple.com/kb/ht1430

How can I disable nagging "Apple-ID" password requests?

Hello,
In order to describe the problem I'm experiencing, I need to say some things in advance. I'm trying to install the same software on each of our 10 iPads. To do so, I install all necessary apps on one selected iPad (from here on "iPad No. 1") and do all in-app-purchases I want to do. On this particular iPad, everything will work fine, but not on the other ones.
Afterwards, I create a backup of iPad No. 1 within iTunes using the Option "Save" (all captions are translated from German and might not match exactly with English captions). Then, all iPads are connected to my computer, one after another. All of them have the same name and are fed with the data from iPad No. 1 using the iTunes option "revert to backup...".
Then, all iPads are supplied with a profile using the "iPhone Configuration Utility". The profile contains wifi access data among settings for password requests as well as permissions to use the iTunes Store and to do in-app-purchases. The last three options must be unrelated to my problem, though, because I tried all possible combinations of those three options (brute force) and changing the configuration to any of those settings did not solve my problem, which I will come to describe in a second.
Until I did some in-app-purchases, the above procedure worked just fine and I wasn't bothered with any password requests. But now, all of my iPads have the following problems - except for iPad No. 1:
The in-app-purchases are registered on all iPads, i.e. I'm not asked to pay again for what I already paid for. Yet, the additional data and program functionality I bought in iPad No. 1 and downloaded on iPad No. 1 is unavailable on all other iPads and needs to be downloaded manually by the push of a button. Note: I do not need to enter any password in order to download the in-app-purchases I already made. But 12 In-App-purchases on 10 iPads makes 120 manual actions for each time I need to reset the devices to the backup I created. That alone makes this procedure a real hassle, and things are getting worse:
All devices except for iPad No. 1 will ask for "password for Apple-ID" regularly, e.g. when the device is switched on after the backup. Since the devices are to be used in an educational environment and were selected as the devices of choice for the mere reason that they are very easy to use without any knowledge of computer equipment, this makes the whole thing completely unusable.
The password requests will stop once I manually downloaded the in-app-purchases.
In other topics, I read that this "password request problem" can be caused by incompletely downloaded software. People have advised others who had this problem to "synchronize" and "submit purchases" within iTunes. But doing those after "revert to backup" didn't help, as expected.
It is correct, though, that the problem seems to be related to incompletely downloaded software. Yet I don't know what I could do to transfer the in-app-purchases from iPad No. 1 to the other devices. Any kind of help would be greatly appreciated, even if it's just a partial solution to what I described.
Best regards,
Jörg

If your two IDs have duplicate contacts, simply turn contact syncing off in one of them. You can do this on your Mac at system preferences > iCloud or on a mobile device at settings> iCloud.

IOS 6.1 on iPhone 5: multiple Apple ID password requests on restart

After installing iOS 6.1 on my iPhone 5 and restarting the iPhone, I was besieged with requests to enter my Apple ID password...6 in a row, one after another. After things calmed down, I turned off the iPhone completely and turned it on again to see what would happen, and got another 6 password requests. This onslaught of requests does not happen on my other two iOS devices, an iPad 2 and a 3rd generation iPad, but it happens every time I power down (not sleep, but power down) and then restart the iPhone.

No. You must sign in with your Apple iD.

Retrieve data/files fro HTTP POST request in On-Demand process

Hello,
I would like to integrate https://github.com/blueimp/jQuery-File-Upload to my APEX 4.2 application inside XE11g. I would like to use this kind of jQuery component, multiple file upload, use Drag & Drop, image resize, size limit, filter extensions etc...
This jQuery component and also others javascript uploaders sends data files to some defined URL. Developer need to build some servlet, php script or something on server side that will read files from HTTP request and stores it somewhere.
Do you know how to do it in APEX? How can I read data from HTTP POST request in PL/SQL? Now I can only call my On-Demand application process from javascript, but I am not able to read any data from HTTP POST in it.
Can I do it in APEX, or using MOD_PLSQL?
I would like to implement this:
1) some javascript uploader will call some URL of my application and sends HTTP POST with data files
2) I will read data files from HTTP POST and store them into database
3) I will create some HTTP response for javascript uploader
Thank you for some tips

I know about that existing plugin " Item Plugin - Multiple File Upload"
But I doesn't work in IE and has only basic features. Licence for commercial use is also needed. I would like to use some existing jQuery plugin. There are many of these plugins with nice features. And only one problem is that I need to create some server side process/servlet/script.. that can recieve HTTP request, get files from it and stores them into DB.

Authenticating a crawler thru a http password request prompt

Similar Messages

Maybe you are looking for