Process sampled data and perform simple calculation to by pass Matlab?

Question

I have files that contain large number current samples (like 300,000 for example). I need them to performing simple calculations (for example to calculate energy).
Before, I used Matlab to load all those samples into array and to do all calculations there.
However... that process is kind of slow. What can I use to simply this and replace Matlab by some script?
Can I use perl or php to perform this operation on set of sample files to spit out final calculation values into new file?
Last edited by kdar (2012-05-11 13:21:05)

Answer 1

I'm surprised no one has suggested numpy and scipy, which are modules for python 2. They have syntax that's similar to mattlab, they're very fast, and python is useful in other circumstances and easy to learn.
I did something similar for spectral data. Here's an example of what I did in python:
#!/usr/bin/env python
#This program goes through rayleigh line data and finde the mean shift
#in nanometers and the standard deviation
import sys, os
import numpy as np
import scipy as sp
import scipy.optimize as op
import time
ray = []
filenames = []
line = 633
def rs(wavelength,laser):
return ((float(1)/laser)-(float(1)/wavelength))*(10**7)
def main(argv): #Goes through a file and finds the peak position of the rayleigh line
f = np.loadtxt(argv).transpose() #opens the file
maxi = np.amax(f[1]) #Finds the value of hte peak of the rayleigh line
intensity = [f[1,i] for i in range(len(f[1]))] #extrants the array into a list
indi = intensity.index(maxi) #Finds the index of the rayleigh line
ray.append(f[0,indi])
filenames.append(str(argv))
# Goes through each file named in the CLI call and applies the main function to it
for filename in sys.argv[1:]:
main(filename)
# Use numpy for some basic calculations
mean = np.mean(ray)
StandardDeviation = np.std(ray)
median = np.median(ray)
variance = np.var(ray)
ramanshift = [rs(ray[i],line) for i in range(len(ray))]
rsmean = np.mean(ramanshift)
rsSD = np.std(ramanshift)
rsmedian = np.median(ramanshift)
rsvariance = np.var(ramanshift)
tname = str(time.asctime())
# Write all calculations to a file
output = open('rayleigh_'+tname+'.dat','w')
output.write('#The files used for this compilation are:\n')
for i in range(len(filenames)):
output.write('#'+filenames[i]+'\n')
output.write('The wavelengths of the Rayleigh line are (in nm):\n')
for i in range(len(ray)):
output.write(str(ray[i])+'\n')
output.write('The raman shifts of the rayleigh line for '+str(line)+'nm are (in rel. cm^(-1):\n')
for i in range(len(ray)):
output.write(str(ramanshift[i])+'\n')
output.write('Mean = '+str(mean)+'nm, or '+str(rsmean)+' rel. cm^(-1)\n')
output.write('Standard Deviation = '+str(StandardDeviation)+' nm, or '+str(rsSD)+' rel. cm^(-1)\n')
output.write('Median = '+str(median)+'nm or, '+str(rsmedian)+' rel. cm^(-1)\n')
output.write('Variance = '+str(variance)+'nm or, '+str(rsvariance)+' rel. cm^(-1)\n')
output.close()
Last edited by Yurlungur (2012-05-13 21:14:54)