PowerShell processing of large CSV very slow - Is there a better way?

Question

Using Powershell I am working through a foreach loop. One of the steps inside the loop is to find a match in a csv file -  
import-csv ".\RateCard.csv" | where-object{$($Item.'From Number') -match "^\+?($($_.Destination))\w*$"}
The foreach loop is processing around 100,000 records for a month time period and completes within a few minutes without the csv lookup. After adding the csv lookup I can see its taking about 5-10 seconds for each lookup. The csv file has 40,000 rows and about
10 columns. 
While I understand that client side processing is going to be slower, and that I should probably look at using SQL, I wondered if there is a faster way to do this. I thought of using a stream reader but couldn't work out how I could do a match lookup.
Any suggestions would be greatly appreciated.
Andrew Morpeth
Lync Server Specialist - Auckland, NZ
Blog - http://www.lync.geek.nz
Twitter @AndrewMorpeth

Answer 1

If you will use multi threading it will be very fast..
I can give you example & syntax, check if it helps..
Start-Job -ScriptBlock { script block}  -ArgumentList
$arg1,$arg2
While((Get-Job -State 'Running').Count -ge 4)
        Start-Sleep -Milliseconds 10
here is the example from one of my scripts:
$compeventid1 = get-content $log1<o:p></o:p>
foreach($evid in $compeventid1){<o:p></o:p>
Start-Job -ScriptBlock {<o:p></o:p>
              $count = 0<o:p></o:p>
                $outlook = new-object
-com outlook.application;<o:p></o:p>
                $ns = $outlook.GetNameSpace("MAPI");<o:p></o:p>
                $inbox = $ns.Folders.Item($($args[0])).folders.Item($($args[1]))<o:p></o:p>
                $messages = $inbox.items<o:p></o:p>
                foreach($message
in $messages){<o:p></o:p>
                $mBody = $message.body<o:p></o:p>
                $mBodySplit = $mBody
-split "Event ID:"<o:p></o:p>
                $toaddress1=$mBodySplit[1]<o:p></o:p>
                $eventid = $toaddress1
-split "Monitoring Category:"<o:p></o:p>
                $eventid = $eventid[0].trim()<o:p></o:p>
        if($eventid -eq $($args[2]))<o:p></o:p>
         {<o:p></o:p>
                $MoveTarget = $ns.Folders.Item($($args[0])).folders.Item($($args[3]))<o:p></o:p>
                [void]$message.Move($MoveTarget)<o:p></o:p>
        $count = $count + 1<o:p></o:p>
        if($count -eq "2")<o:p></o:p>
         {<o:p></o:p>
        break<o:p></o:p>
         }<o:p></o:p>
         }<o:p></o:p>
 <o:p></o:p>
      } <o:p></o:p>
   } -ArgumentList $mbx,$fold,$evid,$tfold<o:p></o:p>
While((Get-Job -State 'Running').Count -ge 4)
        Start-Sleep -Milliseconds 10
Sukhija Vikas
http://msexchange.me