How to delete a random set of indizes from an array as fast as possible?

Question

Hello,
My question sounds a bit like this thread but it isn't...    :-(
http://forums.ni.com/ni/board/message?board.id=170​&message.id=129888&query.id=10689#M129888
When sampling position/force/resistance data I use a time triggered sampling approach. For later analysis I need (among others) force (Y) over position (x).
To get this I calculate the mean force from all time points with equal positions (as time is my common index). That works quite well but with arrays over 300000 points it becomes a bit too slow...
These are my steps so far:
1. Sort the position array
2. Eliminate duplicates from the position array (no array reshape in loop)
3. With the data from 2. get all indices where my original position array has a given value
4. With the indices from 3. get the corresponding force values from the force raw data and calculate the mean
In step 3 I need to search the whole raw position array as many times as there are different position values in itself.
If I could delete all indizes I already found from the array to search that should (in theory) significantly improve performance. On the other hand such an operation would mean a lot of array reshape operations.
What is the best approach to delete the found indizes?
OR
Is there no benefit in deleting the indizes as the array operations to do this need more time than they save?
Thanks!
Sören

Answer 1

Now I'm getting a bit more confused about the app.  If you need a sampling rate higher than the encoder can provide, then then only way I can think of to accumulate multiple readings at the same position is if you have some type of bi-directional cyclical motion.  But if you originally used the encoder as a sampling clock, that seems to imply a unidirectional motion.  Is the motion cyclical or unidirectional?
Knowing that I need to do some similar processing down the road, I did a bit of tinkering today.  The method I described earlier took ~350 msec to process (sort and then calculate averages force per unique position) on 350,000 data points.  A little more tinkering gave me a method that took <100 msec.  The machine was a pretty new test PC with a 2.8 GHz Pentium.  Here's an outline and perhaps I'll be able to scrub the code a bit to post soon.  Hopefully someone will have some even better ideas!
The basic idea is essentially a histogram where each possible integer position value maps to a histogram bin.
1. Based on knowledge of your test equipment, you can know the # of unique positions that can possibly be recorded.  Pre-initialize 2 arrays of this size before starting the main data acquisition.  Both should be full of floating-point 0's.  These will hold (a) count of entries and (b) sum of forces.  Call them the binning arrays.  My test used a size of 8000 possible positions.
2. An auto-indexing For loop goes through the 350,000 integer positions and floating point forces.  The binning arrays enter this loop through shift registers.
3. For each pair, use the integer position to map yourself to the appropriate index of the binning arrays.  Increment that element of the count array and add this iteration's force value to the sum array.
4. After loop completes, divide sum array by count array.  These are your average forces.  (This is why the count is computed as a floating point value.  It was very costly to allow a type coercion from an integer count array into the division function.)  Note that unsampled positions should have 0 counts and produce a NaN result from the division.
There are also ways to calculate a running average in the loop, but I haven't checked (yet) to see if it's faster.  The median you mentioned would need some extra post-loop steps.
I'm sure that if this were a coding challenge, someone would cut the processing time at least in half...  Anyone out there?
-Kevin P.