SH: compare and remove found instances

Question

Forewarning: I'm a scripting beginner and haven't posted in this section before and hope I can describe this properly. Also I'm trying to accomplish this in shell (be it sh, bash, or zsh)...
Yes, well, basically I'm trying to compare output2 to output1 and if there are anything from output2 found in output1 to remove them from output1. So if I'm checking for something I don't want in output1 I'll use output2 as a reference and if output1 contains, say, two values from output2 it will remove them all. Example:
"output1" (good values) | "output2" (bad values)
good mouse | bad cat
good mouse X | bad cat X
bad cat XXX | bad cat XX
good mouse XX | bad cat XXX
bad cat | bad cat XXXX
output1 is compared to a reference of output2 and it finds two instances that aren't supposed to be there, regardless of position in either list. Of course when I compare them it wont be as simple as them all starting with the same letters/words. I'm going to be comparing URLs from two outputs.
I know I can "diff" them like: diff <(output1) <(output2) but like I said they won't be in order and they can't be sorted alphabetically because both lists will usually have different values. I'm just trying to remove a value from one list if it's found in both.
I've found a simple way of checking a single output for duplicates but this only removes the duplicates and not every instance.
awk ' !x[$0]++' file
Also, I hate using temp files. Maybe I could pipe both outputs to fifo? Eh, that's kinda the same thing and not really my main issue.
So, to make things easier to understand I'm going to be dumping two outputs of URLs and comparing them. One list will have "potential URLs" and the other will have "definitely not". By comparing the two I will tell if a "potential URL" is a "definitely not", and so remove it from the "potential" list. The end result will be a single output of "potentials" that were confirmed good (as they weren't in the "definitely not" output). There's more to the grand scheme but I've already figured out how to pre-pend and append things to everyline (or N line/s) and organize the whole mess but I just haven't figured out this part, so any help is appreciated even if it's just a link for me to read. Be gentle
Last edited by milomouse (2010-01-16 11:48:41)

Answer 1

Ooh, a nice cat. :D Yeah, that's what I've been using (also used to be a part of my zsh prompt) but I was trying to find a cleaner solution because I generally abuse grep, cat, awk and pipes. I'm horrible for it. Here's a dirty mock-up of what I'm trying to accomplish:
function mrr() {
local env=/tmp/newmirrors
local fifo1=/tmp/fifo1
local fifo2=/tmp/fifo2
mkfifo /tmp/fifo{1,2}
links -dump 'https://www.archlinux.de/?page=MirrorStatus;orderby=lastsync;sort=1'|\
grep "tp:/"|head -n 10|awk '{print $1}'|sed "s/$/\$repo\/os\/`arch`/" >! $fifo1 &
links -dump 'https://www.archlinux.de/?page=MirrorStatus;orderby=avgtime;sort=1'|\
grep "tp:/"|head -n 10|awk '{print $1}'|sed "s/$/\$repo\/os\/`arch`/" >! $fifo2 &
grep -xvf $fifo2 <$fifo1|grep -vi i686|head -n 5|sed 's/^/Server = /;s/$//' >! $env
rm /tmp/fifo{1,2} &>/dev/null
rankmirrors -n 5 $env >! /tmp/good
rm $env ; sudo mv /etc/pacman.d/mirrorlist /etc/pacman.d/mirrorlist-old
sudo mv /tmp/good /etc/pacman.d/mirrorlist
So, yeah. Basically a crappy attempt on what I believe Xyne's perl Reflector is doing (although I don't know how to read much Perl to tell). I just wanted something I understood.
Anyway, I hate how I use fifo as tmp files but I didn't know how else to do it. Also, I had to add the bit about i686 because some servers on the list are i686 only and specify it in their URL so I had to remove them (since I have x86_64). Oh, and the use of "sudo", blech. I guess it works though. Basically it grabs a list of the most recently updated servers and then weeds out the servers with bad response time (if they're on that list, which a few were when I last tested this). Rankmirrors surprisingly yields different results despite the response-time. I guess I could use ping instead of Rankmirrors but I'll do that in the script, not this function. This is the basic reason I needed to compare them, which I hope to expand upon later. I'm sure there's an easier way to do all this but I simply lack the know-how. Again, this is a temporary function I have in my .zsh* file; the script will be different [if I ever finished it].
edit: typo
Last edited by milomouse (2010-01-16 15:31:18)