Your opinion needed - generating unique ID from a URL

I am currently making a web crawler type program and before rushing straight in and coding something I thought i'd gather a few "expert" opinions so to speak ;)
A crawler obviously takes a URL, downloads it and saves it in a database somewhere. However I need to keep a record of which URLs have been previously crawled - as not to crawl and recrawl the same URL.
Ideally I need to make an algorithm which takes a String URL and outputs a unique ID for that URL - no other URL would be assigned the same ID.
After a while researching I have ran into base64 which could take a URL and output a long String of characters. I have a question here though:
1. Is base64 always going to output UNIQUE IDs for every URL or is there a slight offchance you could have duplicates?
However even if base64 worked and it gave me the ID:
aHR0cDovL3d3dy5qYXZhLmNvbQ==
For the URL:
http://www.java.com
I still have to record it in a database and search for its prescence to determine if the URL has been crawled before.
So my question is:
Is there a way to construct a unique NUMERICAL ID for a URL?
This way, it would be much easier to find the record if all were sorted by their numerical ID number.
Hope I havent confused you, if I have just tell me!
Thanks
-Myles

I need a numerical representation of a URL as it would reduce the time to search for a URl to check if it had been previously crawled.BUNKUM! Sorry, but it is... you're talking about saving pooftenths of second on each database seek... but if you're web-crawler isn't inherently I/O bound (say what 2 to 5 seconds a page) then who's your ISP? and how much do they charge.
IE: Your barking up the wrong tree. Just use an MD5's, keep your "current site" in a hashset... for instance if was crawling http://forum.java.sun.com I'd load a hashset from the DB of that domain... crawl within it, and save the whole shebang back to the DB in one hit, delaying any external links until the end.. then load each of those "sites" and crawl and save them, again delaying any external links... and so on until the end of time in all probability.
And, sorry, you're reinventing the square wheel again... ever heard of jackrabbit?
http://mail-archives.apache.org/mod_mbox/jackrabbit-commits/200502.mbox/%[email protected]%3E
(I love google)
PS: Someone once told me that the internet has a higher probability of surviving a nuclear war than the human race... and if your solar power station also survives then maybe in about a 1,000,000 one cocker-roach might turn to the other cocker-roach and say, "Dad it's finally finished. Can I play Toad Killer 3 now?" ;-)
keith.

Similar Messages

I need to download image from the url and the image is in byte format.

hi
i need to download image from the url
http://www.tidelinesonline.com/mobile/j2me_v1?reqType=imageJoin&imageCount=1&month=1&day=1&year=2008&id=1&imageWidth=230&imageHeight=216&imageDepth=8&imageUnits=feet&imageType=JPG&msisdn=456
first 5 digits will be its length,we need to download except 5 digits.
can any one know how to do that.
thanks in advance
Mraj

hi
i need to download image from the url
http://www.tidelinesonline.com/mobile/j2me_v1?reqType=imageJoin&imageCount=1&month=1&day=1&year=2008&id=1&imageWidth=230&imageHeight=216&imageDepth=8&imageUnits=feet&imageType=JPG&msisdn=456
first 5 digits will be its length,we need to download except 5 digits.
can any one know how to do that.
thanks in advance
Mraj

N73ME Update: Your Opinion Needed

The current firmware version on my mobile is 4.0727.2.2.1
Each time a new firmware was released i really didn't want to upgrade to it bec of the bugs i heard it contained.
And now, with the release of this new firmware 4.0812.4.2.1, i really would appreciate your opinion, is it worth that i upgrade to this fw, or should i stick to mine (knowing that i have no complains about my fw).
Thanks alot in advance

if u dont have anyting to complain about ur current firmware den i would recommend u not to upgrade ur n73 firmware. Nokia have trully destroyed the N73 with these firmware updates for example musci player crashes, camera blue noise. etc
If you find my post useful then click on
Kudos!Nokia N96 (v20.050 / RM-247)
www.shaysoft.net | Competitions!

[noob needs help] Loading image from a URL

Hey everyone. I'm new here. And I'm new at J2ME as well.
I have a simple school project and I have doubts about it. I've searched everywhere on the net yet I could not find the solution. Anyways, I do understand the concept of loading images by putting the files in the "res" folder and use:
try
img = Image.createImage("/burgerimgsmall.jpg");
catch (Exception e)
e.printStackTrace();
But my project requires me to load an image from a URL. How do I go about doing that? If it helps, here's the URL:
http://img.photobucket.com/albums/v703/punkgila/burgerimgsmall.jpg
Thanks guys!

Don't worry, downloading image from http is also simple. Look into Photoalbum demo in wtk2.5

Your opinion needed

I ma doing gcse art and design graphic design in school. I always do my graphics in photoshop but had to do it in firework that day (dont ask!!). Plz tell me ur opinion of it, be honest!! this is my first ever try on fireworks, i never saw or used it before!!

Well, my design style is very different! I like more minimalist looks and I do Web sites that need to appeal to broad audience ranges. I wouldn't want to hamper your vision in any way.
I think your bright pop-art-style colors are a good fit for mp3 player packaging, though. Consider who the people would be that would buy an mp3 player. Many of them are your own age, so designing for yourself is designing for your customer base.
If you're just doing work for class, then you can probably pull images off the Internet, because only your teacher will see it. In general, however, no. In most cases, when someone makes an image (no matter how), they own the copyright and if you take it and do something with it then you've created a 'derivative work" to which the original image creator still owns the copyright. If you plan on entering anything in a contest or selling anything, then your work should be original from the start, or use base imagery you have permission to use.
Some images are "public domain" and available for you to use. Putting an image on the Internet does not make it public domain. The creator of the image must specifically say that he or she is releasing it as a public domain image.
Some reading for you:
http://www.templetons.com/brad/copymyths.html
http://lib.sdstate.edu/GuideoftheMonth/image.html
http://www.netknowhow.ca/nkhcrimg.html
http://www.umuc.edu/library/copy.shtml
And to look for public domain images of New York, here's a Google search:
http://www.google.com/search?hl=en&source=hp&q="public+domain"+image+"new+york"&aq=f&oq=&a qi=g-p1g9
And, all this copyright stuff applies to any artwork you create, too. Nobody can ever take it away from you; you have to release your work (or, if you're working for someone else, the copyright may be theirs).

How to generate unique filenames??

i need to be able to generate unique files from a servlet..
my initial instinct was to use the seesion id as part of the filename, however as this file will be embedded in the responding html, this is not safe, as the user will only have to look at the html source to view a session id value.
i am now considering to use the date/time of the creation of a session as the unique identifier for the file, however this will not work if two sessions can be created at the same time.
my question thefore is if no two sessions can have the same date and time?
if not.. can anyone give me an idea as how to produce unique filenames?

If you are actually creating a file, why not usethe
java.io.File.createTempFile() method?how does that help with prducing a unique value to use
as the name of a file?
with that method i still need to supply the filename
as one of its arguments!
No you don't. You provide a prefix and suffix and it fills in the middle with something guaranteed to be unique, in the directory you specify. Here is the javadoc:
createTempFile
public static File createTempFile(String prefix,
String suffix,
File directory)
throws IOException
Creates a new empty file in the specified directory, using the given prefix and suffix strings to generate its name. If this method returns successfully then it is guaranteed that:
1. The file denoted by the returned abstract pathname did not exist before this method was invoked, and
2. Neither this method nor any of its variants will return the same abstract pathname again in the current invocation of the virtual machine.
This method provides only part of a temporary-file facility. To arrange for a file created by this method to be deleted automatically, use the deleteOnExit() method.
The prefix argument must be at least three characters long. It is recommended that the prefix be a short, meaningful string such as "hjb" or "mail". The suffix argument may be null, in which case the suffix ".tmp" will be used.
To create the new file, the prefix and the suffix may first be adjusted to fit the limitations of the underlying platform. If the prefix is too long then it will be truncated, but its first three characters will always be preserved. If the suffix is too long then it too will be truncated, but if it begins with a period character ('.') then the period and the first three characters following it will always be preserved. Once these adjustments have been made the name of the new file will be generated by concatenating the prefix, five or more internally-generated characters, and the suffix.
If the directory argument is null then the system-dependent default temporary-file directory will be used. The default temporary-file directory is specified by the system property java.io.tmpdir. On UNIX systems the default value of this property is typically "/tmp" or "/var/tmp"; on Microsoft Windows systems it is typically "c:\\temp". A different value may be given to this system property when the Java virtual machine is invoked, but programmatic changes to this property are not guaranteed to have any effect upon the the temporary directory used by this method.
Parameters:
prefix - The prefix string to be used in generating the file's name; must be at least three characters long
suffix - The suffix string to be used in generating the file's name; may be null, in which case the suffix ".tmp" will be used
directory - The directory in which the file is to be created, or null if the default temporary-file directory is to be used
Returns:
An abstract pathname denoting a newly-created empty file
Throws:
IllegalArgumentException - If the prefix argument contains fewer than three characters
IOException - If a file could not be created
SecurityException - If a security manager exists and its SecurityManager.checkWrite(java.lang.String) method does not allow a file to be created
Since:
1.2

Passing parameters from an URL to a WAD Web Template 7.0

Hi guys, I need to pass parameters from an URL to a web template created with WAD 7.0. We need to open a template with certain filters starting from a link button located on an Xcelsius Dashboard. Our customer wants a web template to be opened with the filters defined within the Xcelsius
The idea is to run the template with the filters defined in the URL. I know this was incredibly easy with WAD 3.x, but I didn't find any suitable solutions to use with WAD 7.0.
What is the easiest way to do this with WAD 7.0?
Thanks in advance and regards.
Fabio Cristi

Hi,
It is possible in 7.0, but unfortunately it is a lot more complicated than i 3.x. There are quite a few examples here:
http://help.sap.com/saphelp_nw70/helpdata/en/43/ef05462480025ae10000000a1553f7/content.htm
I used it in a case where I wanted to embed a Web template in a BPS application and pass characteristic filters from the BPS application to the web template - similar to your requirements. Basically you need the following just to restrict one characteristic :o(
&BI_COMMAND_1-BI_COMMAND_TYPE=SET_SELECTION_STATE
&BI_COMMAND_1-TARGET_DATA_PROVIDER_REF_LIST-TARGET_DATA_PROVIDER_REF_1=DP_1
&BI_COMMAND_1-CHARACTERISTICS_SELECTIONS-CHARACTERISTIC_SELECTIONS_1-SELECTIONS-SELECTION_1=SELECTION_INPUT_STRING
&BI_COMMAND_1-CHARACTERISTICS_SELECTIONS-CHARACTERISTIC_SELECTIONS_1-SELECTIONS-SELECTION_1-SELECTION_INPUT_STRING=PDS06
&BI_COMMAND_1-CHARACTERISTICS_SELECTIONS-CHARACTERISTIC_SELECTIONS_1-CHARACTERISTIC=0D_PH2
I could only get it to work with one characteristic and I suspect, that the reason is that the URL simply gets to long (very quickly). You have to repeat the above for every characterstic you want to use :o(
Hth,
Jacob

I have to generate a 4 char unique string from a long value

I got a requirment
I have to generate a 4 char unique string from a long value
Eeach char can be any of 32 character defined has below.
private static final char char_map[] = new char[]{'7','2','6','9','5','3','4','8','X','M','G','D','A','E','B','F','C','Q','J','Y','H','U','W','V','S','K','R','L','N','P','Z','T'};
So for 4 char string the possible combination can be 32 * 32 * 32 * 32 = 1048576
If any one passes a long value between 0 - 1048576 , it should generate a unique 4 char string.
Any one with idea will be a great help.

Well, a long is 64 bits. A char is 16 bits. Once you determine how you want to map the long's bits to your char bits, go google for "java bitwise operators".

Hey guys I was wondering do I need an anti virus software on my mac? I don't think I need it but as a first time mac user I wanted your opinion.

Just wanted to get your opinion.

This comment applies to malicious software ("malware") that's installed unwittingly by the victim of a network attack. It does not apply to software, such as keystroke loggers, that may be installed deliberately by an attacker who has hands-on access to the victim's computer. That threat is in a different category, and there's no easy way to defend against it. If you have reason to suspect that you're the target of such an attack, you need expert help.
All versions of OS X since 10.6.7 have been able to detect known Mac malware in downloaded files. The recognition database is automatically updated once a day; however, you shouldn't rely on it, because the attackers are always at least a day ahead of the defenders. In most cases, there’s no benefit from any other automated protection against malware.
Starting with OS X 10.7.5, there is another layer of built-in malware protection, designated "Gatekeeper" by Apple. By default, applications that are downloaded from the network will only run if they're digitally signed by a developer with a certificate issued by Apple. Applications certified in this way haven't actually been tested by Apple (unless they come from the Mac App Store), but you can be sure that they haven't been modified by anyone other than the developer, and his identity is known, so he could be held responsible if he knowingly released malware. For most practical purposes, applications recognized by Gatekeeper as signed can be considered safe. Note, however, that there are some caveats concerning Gatekeeper:
It doesn't apply to software that comes packaged as an installer. Treat all third-party installers with caution.
It can be disabled or overridden by the user.
It can be bypassed by some third-party networking software, such as BitTorrent clients and Java applets (see below.)
It only applies to applications downloaded from the network. Software installed from a CD or other media is not checked.
For more information about Gatekeeper, see this Apple Support article.
Notwithstanding the above, the most effective defense against malware attacks is your own intelligence. All known malware on the Internet that affects a fully-updated installation of OS X 10.6 or later takes the form of so-called "trojan horses," which can only have an effect if the victim is duped into running them. If you're smarter than the malware attacker thinks you are, you won't be duped. That means, primarily, that you never install software from an untrustworthy source. How do you know a source is untrustworthy?
Any website that prompts you to install a “codec,” “plug-in,” or “certificate” that comes from that same site, or an unknown one, is untrustworthy.
A web operator who tells you that you have a “virus,” or that anything else is wrong with your computer, or that you have won a prize in a contest you never entered, is trying to commit a crime with you as the victim. [Some reputable websites did legitimately warn users who were infected with the "DNSChanger" malware. That exception to this rule no longer applies.]
“Cracked” copies of commercial software downloaded from a bittorrent are likely to be infected.
Software with a corporate brand, such as Adobe Flash Player, must be downloaded directly from the developer’s website. No intermediary is acceptable.
Java on the network (not to be confused with JavaScript, to which it's not related) is always a potential weak spot in the security of any operating system. If a Java web plugin is not installed, don't install it unless you really need it. If it is installed, you should disable it (not JavaScript) in your web browsers. Few websites have Java content nowadays, so you won’t be missing much. This setting is mandatory in OS X 10.5.8 or earlier, because Java in those obsolete versions has known security flaws that make it unsafe to use on the Internet. The flaws will never be fixed. Regardless of version, experience has shown that Java can never be fully trusted, even if no vulnerabilities are publicly known at the moment.
Follow these guidelines, and you’ll be as safe from malware as you can reasonably be.
Never install any commercial "anti-virus" products for the Mac, as they all do more harm than good. If you need to be able to detect Windows malware in your files, use the free software ClamXav — nothing else.

Hi I need this asap... "Java code to generate XML File from XML Schema"

Hi all....
I need this asap... "Java code to generate XML File from XML Schema i.e XML Schema Definition, XSD file".
Thankz in advance...
PS: I already posted in the afternoon... this is the second posting.

take look at :
http://sourceforge.net/projects/jaxme/
this might help...

Help needed to get unique record from an internal table

Hi Everybody,
I have to get unique record from an internal table. i know we can use read statement with key condition .
But the problem is i have to use some relational operators like GE or LE.
eg
read table itab into wa with key width GE itab-widthfrom
                                                   width LE itab-widthto
                                                   machno eq itab-machno.
Its giving me error when I use the operators GE , LE.( I think since it can't find a unique record with those relational
operators in the with key statement)
Is there any other way to get unique record from internal table without using the loop?
Thanks,
Sunny

Using the read statement you will need some kind of loop. For example.
DO.
READ TABLE......
WITH KEY ......
                    READ = SPACE
IF SY-SUBRC EQ 0
TABLE-READ = 'X'.
MODIFY TABLE
ADD 1 TO W_FOUND.
ELSE
EXIT.
ENDIF
ENDDO.
IF W_FOUND EQ 1.
...record is unique.
ENDIF.

U160: Need help finding one and your opinion. Hardware update coming soon?

I really want to pick up an Ideapad u160 after playing around with a x120e for the last week. But, in the US it is nearly impossible to find one.
Lenovo does offer the Core i3 version for $750 (a bit over priced in my opinion) from the website, but only in brown. I would love to get a black on with an i5 or i7.
The only place I could find them was on eBay and that was a refurbished one: http://cgi.ebay.com/LENOVO-Ideapad-U160-12-500GB-4GB-BT-i7-640UM-Ultrathin-/270778848252?pt=Laptops_...
How are Lenovo's refurbished laptops? And I am guessing the u160 would be a huge bump in performance from the x120e.
Finally, there has been no talk of a refresh for the u160 line right? I would hate to buy one and then be left with the old modle.
Thanks!

You don't want to reset all settings .... You have to totally restore the iPad to factory settings which erases the device. After you restore the iPad, you can sync your content back onto the iPad from your iTunes library.
Read this to see what I am talking about. This article talks about the iPhone, but it works the same way with an iPad or an iPod Touch.
http://www.maclife.com/article/howtos/how_recover_your_restrictions_password
Do you sync with iTunes on a regular basis? If yo don't, you can download past purchases as long as you use the same Apple ID that you originally bought the content with and as long as the content is still available in the stores (iTunes, the App Store or the iBooks Store).
When you restore the iPad to factory settings, your son's game progress will be lost as well, so keep that in mind. All app data and settings will be erased when you restore as a new device.
This discussion has some helpful tips on how to save some of the iPad content before you restore as new.
https://discussions.apple.com/thread/3695870?tstart=0
Message was edited by: Demo

Need to generate Unique User IDs in OIM

Hi,
I am working in OIM 9.1 environment right now. I need to generate unique User IDs for all the users who come via self request. My problem is that while creating the user ids, I can search for the existing users in OIM and AD easily, but i am unable to search the users for whom the request has been made but approval is pending. Such user ids are reserved in OIM so if an employee with the same first and last name comes, there might be a conflict which might lead to an error. Can someone suggest any api or query using which I can search for these IDs under request in OIM.

If i use the userid generator as Pre-Insert, there will be a conflict if two similar named users request at the same time (or if one of the request is yet to be approved).
I am getting the below error while creating user id as Pre-Insert. What is the correct way of getting this done? Please suggest.
012-11-20 19:25:41,787 INFO [STDOUT] Running CREATEUSERINOIM
2012-11-20 19:25:41,788 INFO [STDOUT] Target Class = com.pldt.adapter.createuser.CreateUserInOIM
2012-11-20 19:25:41,788 INFO [STDOUT] ***********************Inside Create User****************************
2012-11-20 19:25:41,805 INFO [STDOUT] =====================Before Date Insert===============
2012-11-20 19:25:41,805 INFO [STDOUT] =====================hiredate===============1970-01-01
2012-11-20 19:25:41,805 INFO [STDOUT] ***************************section***************XTNL Active
2012-11-20 19:25:42,647 INFO [STDOUT] Running DISABLEUSERINOIM
2012-11-20 19:25:42,649 INFO [STDOUT] Target Class = com.pldt.adapter.disableuser.DisableUserInOIM
2012-11-20 19:25:42,649 INFO [STDOUT] Entered Method retrieveUtility in SelectApprovalUser class
2012-11-20 19:25:42,662 INFO [STDOUT] *************************************Inside disableOIMUser**********************
2012-11-20 19:25:42,662 INFO [STDOUT] *************************************2**************
2012-11-20 19:25:42,663 INFO [STDOUT] ******************tcUserIntf**********Thor.API.Operations.tcUserOperationsClient@5fd3131f
2012-11-20 19:25:42,663 INFO [STDOUT] *************usrkey**********713
2012-11-20 19:25:42,664 INFO [STDOUT] *************usrkey**********XTNL Active
2012-11-20 19:25:42,664 INFO [STDOUT] ********************employeeGroup check Value*************XTNL Active
2012-11-20 19:25:42,664 INFO [STDOUT] returning False
2012-11-20 19:25:42,797 INFO [STDOUT] ***********************User Created Succesfully****************************
2012-11-20 19:25:42,848 INFO [STDOUT] **************************Employee Group**********XTNL Active
2012-11-20 19:25:44,199 INFO [STDOUT] Running CREATEUSERINOIM
2012-11-20 19:25:44,200 INFO [STDOUT] Target Class = com.pldt.adapter.createuser.CreateUserInOIM
2012-11-20 19:25:44,200 INFO [STDOUT] ***********************Inside Create User****************************
2012-11-20 19:25:44,212 INFO [STDOUT] =====================Before Date Insert===============
2012-11-20 19:25:44,213 INFO [STDOUT] =====================hiredate===============1970-01-01
2012-11-20 19:25:44,213 INFO [STDOUT] ***************************section***************XTNL Active
2012-11-20 19:25:44,255 ERROR [XELLERATE.SERVER] Class/Method: tcUSR/verifyUserLogin Error :User Loginid is duplicate.
2012-11-20 19:25:44,258 ERROR [XELLERATE.SERVER] Class/Method: tcUSR/eventPreInsert Error :User login is not correct.

Buying a Macbook pro for the first time and i need your opinion

Hello everyone! I'm buying a Macbook pro for the first time and i need your opinion. I'm a Web Developer / Graphic Designer, and i need this macbook for my work. Currently i have these 2 options:
1) Macbook Pro with Retina (13")
128GB SSD, 2.5GHz Intel i5 Processor, 8GB Ram
2) Macbook Pro i7 (15")
Intel i7 2.6 Processor, 8GB Ram, 750GB HD
So my dilema is. Do i go for the newest 13" Macbook Pro Retina & SSD drive or do i go for the 15" Macbook Pro i7 non-Retina, non-SSD hard drive?
Help this first time Macbook buyer (me) to decide which model to choose

Injektil wrote:
Hello everyone! I'm buying a Macbook pro for the first time and i need your opinion. I'm a Web Developer / Graphic Designer, and i need this macbook for my work. Currently i have these 2 options:
1) Macbook Pro with Retina (13")
128GB SSD, 2.5GHz Intel i5 Processor, 8GB Ram
2) Macbook Pro i7 (15")
Intel i7 2.6 Processor, 8GB Ram, 750GB HD
So my dilema is. Do i go for the newest 13" Macbook Pro Retina & SSD drive or do i go for the 15" Macbook Pro i7 non-Retina, non-SSD hard drive?
Help this first time Macbook buyer (me) to decide which model to choose
Your 2nd choice
2) Macbook Pro i7 (15")
Intel i7 2.6 Processor, 8GB Ram, 750GB HD
is user upgradable. The Retina is not.
No matter what for your needs I would max out the 16GB of RAM

Why do v need 2 generate new smartform from scratch when v have a standard

hi experts
why do v need 2 generate new smartform from scratch when v have so many standard smartforms can any one explain in detail the scenario where v have 2 do that
thanks
bhanu

Hello Bhanu,
Standard smartforms are for standard SAP applications.
Every client will customise standard SAP to suit their own requirements. So if there is a requirement specific to the cleint, they can ask us to develop a custom Smartform.
Award points if found useful.
Regards
Indrajit.

Your opinion needed - generating unique ID from a URL

Similar Messages

Maybe you are looking for