How to download a local mirror of public yum, public-yum-downloader.sh

Question

Hello there
I did write an script to create a local mirror of public-yum.oracle.com, it now includes the errata and security bug fixes information._
First of all, thanks for giving a try to the script.
The script can be located at:
https://github.com/kikitux/public-yum-downloader
Direct RAW access:
https://raw.github.com/kikitux/public-yum-downloader/master/public-yum-downloader.sh
Download as
# wget https://raw.github.com/kikitux/public-yum-downloader/master/public-yum-downloader.sh
The hierarchy is 100% the same as what is on public-yum
The script can take several argumentas, like -P for the OS directory, and --url for where the same path will be public, so you can put the mirror in a different path
example, I have my own repo in /u02/stage/ and is shared like http://mirandaa00/stage
on my apache I have
Alias /stage "/u02/stage/"
<Directory "/u02/stage/">
Options Indexes MultiViews FollowSymLinks
AllowOverride None
Order allow,deny
Allow from all
</Directory>
In that way, I have everything I want in my own path.
When you use the url option, the script will create a local-yum-ol6.repo file with the url you gave, with GPG enabled, so you can be sure nothing wrong will happen in the middle
I use this script it this way
as root, i have /root/bin/dl.sh with this content
~/bin/public-yum-downloader.sh -P /u02/stage/ -p http://proxy:3128 -R 6.latest --url http://mirandaa00/stage -l /u02/stage/repo/OracleLinux/OL6/
~/bin/public-yum-downloader.sh -P /u02/stage/ -p http://proxy:3128 -R 5.latest --url http://mirandaa00/stage -l /u02/stage/repo/OracleLinux/OL5/
~/bin/public-yum-downloader.sh -P /u02/stage/ -p http://proxy:3128 -R 4.latest --url http://mirandaa00/stage -l /u02/stage/repo/EnterpriseLinux/EL4/
~/bin/public-yum-downloader.sh -P /u02/stage/ -p http://proxy:3128 -R 6.4 --url http://mirandaa00/stage -l /u02/stage/repo/OracleLinux/OL6/
~/bin/public-yum-downloader.sh -P /u02/stage/ -p http://proxy:3128 -R 5.9 --url http://mirandaa00/stage -l /u02/stage/repo/OracleLinux/OL5/
~/bin/public-yum-downloader.sh -P /u02/stage/ -p http://proxy:3128 -R 4.9 --url http://mirandaa00/stage -l /u02/stage/repo/EnterpriseLinux/EL4/
~/bin/public-yum-downloader.sh -P /u02/stage/ -p http://proxy:3128 -R 4.8 --url http://mirandaa00/stage -l /u02/stage/repo/EnterpriseLinux/EL4/
~/bin/public-yum-downloader.sh -P /u02/stage/ -p http://proxy:3128 -R 6.UEK --url http://mirandaa00/stage
~/bin/public-yum-downloader.sh -P /u02/stage/ -p http://proxy:3128 -R 5.UEK --url http://mirandaa00/stage
~/bin/public-yum-downloader.sh -P /u02/stage/ -p http://proxy:3128 -r ol6_addons --url http://mirandaa00/stage
~/bin/public-yum-downloader.sh -P /u02/stage/ -p http://proxy:3128 -r el5_addons --url http://mirandaa00/stage
~/bin/public-yum-downloader.sh -P /u02/stage/ -p http://proxy:3128 -r el5_oracle_addons --url http://mirandaa00/stage
~/bin/public-yum-downloader.sh -P /u02/stage/ -p http://proxy:3128 -r ol6_playground_latest
the -l will look on that path to find the rpm, useful for example if you have a dvd and you want to use as initial cache
I do run my commands in that way as when 5.9 came out, I had a lot of those rpms in 5.8 or 5 latest, rite?
Worst thing that could happen, is the rpm is not there, and will have to download, but if it's there will copy it
for UEK and addons those are unique rpm, so I don't use -l
for the playground, that are the new kernel based on 3.x directly, i don't use --url, as I don't wat the script to enable that repo, but I do want to download what that channel have
so, for known versions 6.0 to 6.4 you can use -R 6.n or even -R 6.UEK
for other repos you can pass the name as -r repo
Regarding the OVM3, the OVM3 is not on the repo, so I don't use my script for that, however, you can use the tools your self
mkdir -p /u02/stage/repo/OracleVM/OVM3/latest/x86_64/repodata/.cache
and create a repo file
cat /u02/stage/public-yum-ovm3.repo
[ovm3_latest]
name=Oracle Linux $releasever Latest (x86_64)
baseurl=http://public-yum.oracle.com/repo/OracleVM/OVM3/latest/x86_64/
gpgkey=http://public-yum.oracle.com/RPM-GPG-KEY-oracle-el5
gpgcheck=1
enabled=1
Then, you can download what is there as:
http_proxy=http://proxy:3128 yumdownloader -c /u02/stage/public-yum-ovm3.repo --destdir=/u02/stage/repo/OracleVM/OVM3/latest/x86_64/ '*'
createrepo -v -c /u02/stage/repo/OracleVM/OVM3/latest/x86_64/repodata/.cache /u02/stage/repo/OracleVM/OVM3/latest/x86_64
Please, take note the yumdownloader use --destdir=/path  then SPACE, then what you want to download, as we want a mirror, space '*'
any question, here, or feel free to mailme at [email protected]
if you have time, check http://kikitux.net
Alvaro.
Edited by: Alvaro Miranda on May 6, 2013 9:13 PM

Answer 1

Keep it running, time by time it takes FOR Ever
The good thing is the script already downloaded the repo file and the GPG key, so internet is working.
Where you tell me is waiting for ever, is downloading the metadata of the files, and doing the list of packages and dependencies to download.
then, with that list, it will use wget to download the rpm files
On /var/tmp/public-yum-downloader/ it will be leaving some files that you can use to check what's doing, a folder for x86_64 and i383 to have a local copy of the metadata
/var/tmp/public-yum-downloader/list.log will show is the output of yum-downloader and then the output of wget
I don't think your download will be slower than mine.. I am on New Zealand.. :D
Alvaro.