Automatise data export from website
Hello,
I stuck with an issue regarding data export from scopus.com. I have 61 keywords for which I want to export (as csv) number of documents by year, country, document type, subject area. Doing that by hand would be extremely tedious. Thus my question if this can be automatised. Unfortunately scopus does not provide an API to access the data directly.
So, the process to be automatised would be:
1. Open http://www.scopus.com/home.url
2. Enter the keyword in "search for"
3. Click on "search"
4. Click on "Analyze results"
5. Click on tab "year" and "export". Save the file
6. Click on tab "country" and "export". Save the file
6. Click on tab "document type" and "export". Save the file
6. Click on "subject area" and "export". Save the file
Do programs exist that help me retrieving the data?
Thanks!
You could achieve your goal with bash and wget with some combination of --save-cookies --post-data.
But be aware that scopus is run by Elsevier, whom many view as the most evil of the publishing companies. Think twice before doing something that may annoy them.
Similar Messages
-
Data export from unicode SAP to AIX file shows strange characters in vi
Hello SAP experts,
We have a SAP ECC 6.0 system, Unicode. When we export data from the SAP system to a textfile on AIX and we do a vi on the file the character ë is printed as: ë
We installed the bos.loc.utf.NL_NL package on AIX to no avail.
Any insights you can offer will be much appriciated.
Regards
Dirk VisserThanks for your explanation. It helped me understand better the way ODI works.
Does the pattern has a period '.' in front of it too? I am trying to run a job to load data from flat file to a RDBMS table and one of the steps in which I am getting error is
Code:
create table .C$_0TRG_SALES_PERSON
C1_SALES_PERSON_ID NUMBER(8) NULL,
C2_FIRST_NAME VARCHAR2(80) NULL,
C3_LAST_NAME VARCHAR2(80) NULL,
C4_DATE_HIRED VARCHAR2(80) NULL
Error message:
903 : 42000 : java.sql.SQLException: ORA-00903: invalid table name
I tried running the above piece of code directly on TOAD and I got the same error message - Invalid table name ?
Any idea why the period is getting pre-fixed ? Anyway to remove this ?
Appreciate your help. -
I have created and validated my transformation file, which takes two dimensions and concatenates them. For example Company and ProfitCenter, in the mapping section:
U_Entity=Company+ProfitCenter.
Where my company = 5110 and my profit center = SVS8110
The records are validated and in my output file I can see 5110SVS8110 and the associated transaction data.
However, in the *Conversion section of the Transformation file I reference the conversion file
U_Entity=U_EntityConv.XLS
Within the Conversion file, in the internal column I have 5110SVS8110, and in the external column I have UKSERVCO.
Both files are validated, but when I run the export transaction data package, the output file doesn't include UKSERVCO
In BPC 420 training it says *Conversion can be used to convert external keys from the flat file to internal keys. But can it convert internal keys to external keys in the flat file? If so, what do I need to do here?
Thanks.Hi,
have you tried to invert the column values?
Kind regards
Roberto -
Data Export from Master detail
Hi All,
How to export data from master detail region.
Thanks in advance.
Regards
MSHi MS,
Please use following code
OAExportBean expBean =(OAExportBean)webBean.findChildRecursive("Export");
// set the exportDetails for this Master/Detail region.
expBean.setExportDetails(oaPageContext,"masterRegionName","detailRegionName", "masterVuName", " detailVuName");
Regards,
Reetesh Sharma -
Wrong data export from ID to PS
Hallo,
I have a strange problem pasting placed eps images in ID back to PS.
If I mark (correct name, path and image look) placed eps images in ID CS4 and paste them into a new file at PS wrong images show up! They exist in the same document but are other than marked. But in the created PDF file everything is OK.
On the other hand, if I open the same created PDF file with IL and mark the same images and past them to PS the correct images are shown???? What can be the reason?
Thanks for any comment.
Regards EdzardBy "mark" do you mean select?
Presuming you do, you should not attempt to copy/paste the image from ID to Photoshop. Instead select "edit original" from the context menu or the links panel. That will open the selected image. You may need to use "edit with...) instead if the wrong editing application opens using edit original.
I really don't know why the wrong data would be opening in Photoshop. Is the correct image highlighted inthe links panel? If so, is this happening in all documents, or just a particualr file? You might want to try the links listed in the "Troubleshooting 101" box at the top of the discussions page, up a level from here.
It may also have to do with .eps which is an archaic format for working in ID and is known to cause problems from time to time. Saving as .psd or .tif instead should fix that sort of issue. -
Data Export From InDesign CS6 to Excel Please Help
Hi!
I need to export text from a 416 page catalog into an excel document. The text is formatted with paragraph and character style sheets, but the text frames do not link together. Or is there a way to export the text from InDesign with the page number into a text file, then the text file into an Excel document. Thank you so much for your help!That is not possible. You're going from a "smart" layout application to a dumb one.
I don't think you could even place it as a PDF file of the text into Word and have it work. -
Arabic support in isqlplus after import export from 7.3.4 to 10.2.0.3
CANNOT DISPLAY ARABIC IN ISQL*PLUS(JUNK). APPLICATION WORKS FINE WITH ARABIC
with Taret DB char set to US7ASCII the data is always junk old data +new inserted arabic data inserted thro isqlplus or sqlplus.
with AR8MSWIN1256 the old data(imported in to new db from old 7.3.4 db export dump) is displayed as junk (what ever in the db) and after creating a new table data inserted, the arabic display is fine.
How to overcome the problem of displaying old data? we have lots of data for migration.
ThanksSmall correction
In DB with Char set to AR8MSWIN1256.......
Old data exported from 7.3.4 and imported in new 10.2.0.3 db is showing arabic char as junk
But new table created and data inserted as arabic is displaying properly.
What has to be done such that old imported data displays arabic.
We are using isqlplus and sqlplus to query the data
Thanks -
BPC10 - Data manager package for dimension data export and import
Dear BPC Expers,
Need your help.
I am trying to set up a data manager package for first time to export dimension - master data from one application and import in another application ( both have same properties) .
I created a test data manager package from Organize > add package > with process chain /CPMB/EXPORT_MD_TO_FILE and Add
In the advance tab of each task there are some script logic already populated. please find attached the details of the script logic written under each of the tasks like MD_Source, concvert and target .
I have not done any chnages in the script inside the task .
But when i run the package , I have selected a dimension 'Entity' but in second prompt ,it ask for a transformation file , and syatem autometically add the file ... \ROOT\WEBFOLDERS\COLPAL\FINANCE\DATAMANAGER\TRANSFORMATIONFILES\Import.xls
I have not changed anything there
in the next prmpt , it ask for a output file ..and it won't allow me enter the file name .....i
Not sure how to proceed further.
I shall be greatfull if someone guide me from your experiance how to set up a simple the data manager package for master data export from dimension . Should I update the transformation file in the script for import file and output file in the advance tab. how and what transformation file to be created and link to the data manager package for export / import .
What are the steps to be executed to run the package for exporting master data from dimension and import it another application .
Thanks in advance for your guidance.
Thanks and Regards,
Ramanuj
=====================================================================================================
Detals of the task
Task : APPL_MD-SOURCE
(DIMENSIONMEMBER, %DIMENSIONMEMBERS%, "Please select dimension", "Please select members", %DIMS%)
(TRANSFORMATION,%TRANSFORMATION%,"Transformation file:",,,Import.xls)
(OUTFILE,,"Please enter an output file",Data files (*.txt)|*.txt|All files(*.*)|*.*)
(RADIOBUTTON,%ADDITIONINFO%,"Add other information(Environment,Model,User,Time)?",1,{"Yes","No"},{"1","0"})
(%TEMPNO1%,%INCREASENO%)
(%TEMPNO2%,%INCREASENO%)
(/CPMB/APPL_MD_SOURCE,SELECTION,%DIMENSIONMEMBERS%)
(/CPMB/APPL_MD_SOURCE,OUTPUTNO,%TEMPNO1%)
(/CPMB/EXPORT_MD_CONVERT,INPUTNO,%TEMPNO1%)
(/CPMB/EXPORT_MD_CONVERT,TRANSFORMATIONFILEPATH,%TRANSFORMATION%)
(/CPMB/EXPORT_MD_CONVERT,SUSER,%USER%)
(/CPMB/EXPORT_MD_CONVERT,SAPPSET,%APPSET%)
(/CPMB/EXPORT_MD_CONVERT,SAPP,%APP%)
(/CPMB/EXPORT_MD_CONVERT,OUTPUTNO,%TEMPNO2%)
(/CPMB/FILE_TARGET,INPUTNO,%TEMPNO2%)
(/CPMB/FILE_TARGET,FULLFILENAME,%FILE%))
(/CPMB/FILE_TARGET,ADDITIONALINFO,%ADDITIONINFO%))
Task : EXPORT_MD_CONVERT
(DIMENSIONMEMBER, %DIMENSIONMEMBERS%, "Please select dimension", "Please select members", %DIMS%)
(TRANSFORMATION,%TRANSFORMATION%,"Transformation file:",,,Import.xls)
(OUTFILE,,"Please enter an output file",Data files (*.txt)|*.txt|All files(*.*)|*.*)
(RADIOBUTTON,%ADDITIONINFO%,"Add other information(Environment,Model,User,Time)?",1,{"Yes","No"},{"1","0"})
(%TEMPNO1%,%INCREASENO%)
(%TEMPNO2%,%INCREASENO%)
(/CPMB/APPL_MD_SOURCE,SELECTION,%DIMENSIONMEMBERS%)
(/CPMB/APPL_MD_SOURCE,OUTPUTNO,%TEMPNO1%)
(/CPMB/EXPORT_MD_CONVERT,INPUTNO,%TEMPNO1%)
(/CPMB/EXPORT_MD_CONVERT,TRANSFORMATIONFILEPATH,%TRANSFORMATION%)
(/CPMB/EXPORT_MD_CONVERT,SUSER,%USER%)
(/CPMB/EXPORT_MD_CONVERT,SAPPSET,%APPSET%)
(/CPMB/EXPORT_MD_CONVERT,SAPP,%APP%)
(/CPMB/EXPORT_MD_CONVERT,OUTPUTNO,%TEMPNO2%)
(/CPMB/FILE_TARGET,INPUTNO,%TEMPNO2%)
(/CPMB/FILE_TARGET,FULLFILENAME,%FILE%))
(/CPMB/FILE_TARGET,ADDITIONALINFO,%ADDITIONINFO%))
Task : FILE_TARGET
(DIMENSIONMEMBER, %DIMENSIONMEMBERS%, "Please select dimension", "Please select members", %DIMS%)
(TRANSFORMATION,%TRANSFORMATION%,"Transformation file:",,,Import.xls)
(OUTFILE,,"Please enter an output file",Data files (*.txt)|*.txt|All files(*.*)|*.*)
(RADIOBUTTON,%ADDITIONINFO%,"Add other information(Environment,Model,User,Time)?",1,{"Yes","No"},{"1","0"})
(%TEMPNO1%,%INCREASENO%)
(%TEMPNO2%,%INCREASENO%)
(/CPMB/APPL_MD_SOURCE,SELECTION,%DIMENSIONMEMBERS%)
(/CPMB/APPL_MD_SOURCE,OUTPUTNO,%TEMPNO1%)
(/CPMB/EXPORT_MD_CONVERT,INPUTNO,%TEMPNO1%)
(/CPMB/EXPORT_MD_CONVERT,TRANSFORMATIONFILEPATH,%TRANSFORMATION%)
(/CPMB/EXPORT_MD_CONVERT,SUSER,%USER%)
(/CPMB/EXPORT_MD_CONVERT,SAPPSET,%APPSET%)
(/CPMB/EXPORT_MD_CONVERT,SAPP,%APP%)
(/CPMB/EXPORT_MD_CONVERT,OUTPUTNO,%TEMPNO2%)
(/CPMB/FILE_TARGET,INPUTNO,%TEMPNO2%)
(/CPMB/FILE_TARGET,FULLFILENAME,%FILE%))
(/CPMB/FILE_TARGET,ADDITIONALINFO,%ADDITIONINFO%))
================================================================================1. Perhaps you want to consider a system copy to a "virtual system" for UAT?
2. Changes in QAS (as with PROD as well) will give you the delta. They should ideally be clean... You need to check the source system.
Another option is to generate the profiles in the target system. But for that your config has to be sqeaky clean and in sync, including very well maintained and sync'ed Su24 data.
Cheers,
Julius -
Unable to find out remaining data from website or My Verizon - anyone else?
For the last few days I have been unable to check data usage from either the Verizon website by logging in or from My Verizon on my phone - it gives a very useless "readout" but there's no way to determine how much data I have. Guess this has been since the prepaid "system upgrade" a week ago. Is this the case with everyone and does anyone have ANY idea when this will be resolved? I just can't face calling customer service.
A work-around is to call #3282 from your prepaid phone - it does tell you remaining data. I have to assume that it's correct.I'm in the same boat. I switched from postpaid to prepaid and it just shows my postpaid acct. Myverizon app is usless I can't even log in or register with it...
-
I have made a burn folder with photos exported from I-photo. It now shows in information, that the date created and modified is different from the original digitized date. How can I get the original date to show in the info from Finder?
The Finder reports File information. The date and time of the photo are in the Photo's Exif metadata. The Finder has no awareness of this. All photos apps on any system do.
Regards
TD -
Is it possible to Extract the Data from Website?
Dear Experts,
Is it possible to Extract the Data from Website and Upload it in the SAP Table?.It is very Urgent.Please help me.
Thanks & Regards,
Ashok.Hi Friend,
Generaly if it is one time activity you can do it manually by copy paste in excel.
But if it is weekly or monthly activity then you have to ask website provide to provide data in excel or CSV file and upload it into SAP.
I have a situtation like this, and they (Rapaport price) provide it in CSV format with defined columns.
So it is better to raise the issue to website provider.
Regards
Krishnendu -
How to retrieve the data from Website and Upload it in SAP table?
Dear ABAPers,
I want to retrieve the data from website and upload the same in SAP Database Table is that possible.Please help me.It is very Urgent.
Thanks & Regards,
Ashok.Dear Abhishek,
Thanks for your reply.But my requirement is not met.
If i execute the program it should retrieve the data from particular website.
Thanks & Regards,
Ashok. -
Images stripped from Camera Data after Export from Lr
Hi,
I've just opened images in jpg format in Ps. The images were exported from Lr and for some reason all the Camera Data is gone...
I went through all the settings in Ps and Lr and didn't find anything...
I'm enclosing a screenshot.Hi, this is a similar to an earlier post and here is my reply:
In the Export Dialog Box, under Metadata, make sure that you checked the box for "Write Keywords as Lightroom Hierarchy", in this way all your keywords will be carried over to Photoshop. Conversely, if you do not want your keywords or other metadata to show in Photoshop, then check the box for "Minimize Embedded Metadata". I hope this helps.
Arnel
www.arnelgarcia.com -
Execution date in Excel export from a query
Dear all,
I am wondering if it is possible to have the query execution date in an exported excel.
Case: When working in a query it should be possible to export the query to excel. However I am missing the execution date of the query and the last successful data load (which is visible in the webquery).
Anyone has an idea how to achieve that?
Thank you in advance,
AndreasHey Guys,
Just wanted to point out that I know how to do this with a webapplication but I would need that within an excel export from an ordinary web query.
Could someone help out here?
Thanks in advance,
Andreas -
Data Export to Oracle table from essbase issue
Hello,
i am using a data export calc script to load data from essbase into an oracle table. We have Essbase 11.1.2.1 on windows 2008 64 bit R2 server. I have an ODBC system DSN created for this job.
However, when i launch this process i get a message in the log "Cannot read SQL driver name for [Backup Exec Catalogs] from [ODBC.INI]"
i have checked the ODBC.ini file also in C:\Windows directory and that file contains the connection entries...
Any thoughts as to why i am getting this error in one server whereas the same process on SIT server is working fine.
thanks...please restart the application and try again . Check the application log if any details are there
Maybe you are looking for
-
CDs import but don't show in library
Importing CDs got very weird a couple weeks ago. It may coincide with my signing up with Match, not sure about the exact timing. Here's what happens: I insert a CD and it immediately begins to import. When it finishes there is no sign of it. The song
-
Update all rows in a table which has 8-10 million rows take for ever
Hi All, Greetings! I have to update 8million rows on a table. Basically have to reset the batch_id with the current batch number. it contains 8-10 million rows and i have tried with bulk update and then also it takes long time. below is the table str
-
Ok, I can't get this page: http://web.mac.com/jsimoneau/iWeb/Site/Photos%204.html to work in Explorer on windows because I dragged that top shape behind my logo, off the page so it would only have rounded corners on the bottom. It looks fine in Safar
-
I'm creating a Flash Form and I have a group of 4 checkboxes, and the value part of the input statement is as follows: <cfif Isdefined('Form.whatever1')>1<cfelse>1</cfif> <cfif Isdefined('Form.whatever2')>1<cfelse>1</cfif> <cfif Isdefined('Form.whate
-
dear all, is it possible to write sql statement in ECC without abap? e.g. test a sql statement query with any abap required?