Sine waveform data Compression Techniques

Hi Engineers,
I am looking for some techniques or algorithms to compress the Sine Wave waveform data.
(Some of the changes I already done like DBL to SGL format and 16 bit integers )
I can't effort the sample loss.
Thanks and Regards
Himanshu Goyal
Thanks and Regards
Himanshu Goyal | LabVIEW Engineer- Power System Automation
Values that steer us ahead: Passion | Innovation | Ambition | Diligence | Teamwork
It Only gets BETTER!!!

Himanshu,
simple mathematic for binary files:
30 channels a 10KS/s presumably double => 30*10.000*8 Bytes/s = 2.400.000 Bytes/s (roughly 2MB/s).
Running the application for 1 hour should result in 2MB/s*60*60 = 7.2GB.
So the file you are getting is the most condensed version of data without lossing information.
Loss of information will bring that down to less amount of space needed, but you have to consider which information you want to discard.
Possible ways:
1) Convert all data to Single: You will lose information, eventually cutting off values if the values are very big or very small. On the other hand, you cut down the space to 1/2 (3.6GB/h)
2) Averaging: Calculate the average for several values. This is ok for good oversampling (>1000) and small numbers (<50) esp. when the signal has lots of noise. The space needed is cut down by the amount of values you build the average on. Please note that you cannot use a moving average (which is in fact a simple filtering method).
3) Calculate a form-fit function for packages of the signal and store the parameters for the given function: Best compression, but will lose nearly all information of the waveform and introducing uncertainties by the form fit function (increasing errors). In addition to that, packages might have steps to one another since the form fit function will not result in a continuous function without steps.
There are more methods for sure, but those are the most basic and common that i can think of.
hope this helps,
Norbert
CEO: What exactly is stopping us from doing this?
Expert: Geometry
Marketing Manager: Just ignore it.

Similar Messages

Data Compression express VI chops off last waveform element

While using the data compression express vi, I noticed the vi truncates the array of waveforms by removing the last element.
When I log this to an excel file, it puts a blank row into it. I ran this VI twice which appended two runs in teh same log file to show it.
What is an easy way to delete this last element out of the signal array?
See the graphs last sample on the x axis of the vi I attached. Element at x=1.0 has been deleted on the graph thus resulting in a blank row in my excel file.

Hi Id,
I think this happens because it is using the data that comes after the point to average that point, though I am uncertain. I will have to look into it further.

Is data compression all automatic? Or does manual steps occur in HANA Studio

Hi all,
I'm new to HANA and have been leveraging SCN in a BIG way to answer my questions (thanks everyone for your contributions). This is my first post as I was unable to find an answer to my current question .
I've been reading up on data compression in HANA and I learned that there are different techniques, such as Dictionary, Cluster, and Run-length encoding.
My Question is: Is the compression automatic? Or does it need to be modeled within HANA Studio? Let's use Dictionary Encoding as an example. Are there algorithms in place within HANA that will automatically aggregate values so only distinct values remain? Will the attribute vector and inverted indexed tables automatically be created?
Just as some background, this is what I am basing my question on:
http://www.agilityworks.co.uk/our-blog/demystifying-the-column-store-%E2%80%93-viewing-column-store-statistics-in-sap-ha…
Thank you!
Kyle

Hi Justin,
you are right, the compression is related to the delta merge - and therefore, as long as delta merges happen automatically, compression will also happen automatically.
SAP HANA has two compression stages on the column store: the first - and typically dominating one - is the implicit compression obtained from using a data dictionary for all column-store columns.
The second stage is often called "sparse compression" - it offers additional compression on top of the dictionary compression, with several available algorithms. During the "optimize compression" run, the most appropriate compression algorithm is chosen for each column (and some columns may not be sparse-compressed at all, because it would not bring a benefit).
The optimize compression run does not happen with every delta merge. Instead, it will be performed with the first delta merge on a given table, and then only if the data content of the table has changed significantly (typically, if I remember correctly, the system waits until the number of records has doubled. Note that the optimize compression run may _change_ the compression type chosen for a given column. The sparse compression itself will be done with every delta merge, using the algorithms selected in the past optimze compression.
If you have switched off automatic delta merging globally, or if you have disabled automatic delta merging for a given table, there will also be no automatic compression (in the entire system or on the given table). BW on HANA uses the smart merge feature, so that in a standard BW on HANA, automatic compression will happen (with the timing of the delta merge being decided in a cooperation between BW and HANA).
Best,
Richard

Sine waveform to double indicator (not array)

Hi!
I am using the sine waveform.vi, and I get a waveform, as standard output. If I use "Get waveform components". I get a 1D array from "Y". . What I want is to use the current amplitude value of the sine waveform as a double data type. How is this possible? If I use an index array, I can get all the array components, but I don't see how that is going to solve my problem.
Thanks for your help!
Is my avatar the dog from the fraggle rock?

Waveform is combination of DataArray, dt and timestamp usually. So as you use "get wfm component" you will get array of Y data. This is what expected also as waveform is not of single scalar value.
But to see the last recent element in your array you can use index array function.If you know the number of element in your array. If you don't know then use Array size function to know size and substract one from it then put it as index of array to get last value inside your array.
In my personal experience on any waveform one interest area are amplitude, phase, frequency and sampling info etc.. thease are details which you are using to generate.
So please varify once again what you want..
- HS

Devo generare una sine waveform con una certa fase iniziale ma che sia reentrant

devo generare una sine waveform con una certa fase iniziale ma che sia reentrant, in quanto sto simulando una digital down conversion e genero per ogni simbolo la moltiplicazione dei campioni del simbolo trasmesso con un numero di campioni uguali di una sinusoide con una certa frequenza. Quindi per ogni simbolo devo far partire la sinusoide con la fase con cui ha terminato l'iterazione precedente. Inoltre, dato che sto simulando una 16 QAM ho la necessità di creare due portanti, una seno ed una coseno, entrambe rientranti, ma con fase iniziale sfasata di 90 ° tra loro. Però, il vi SineWaveform ignora la phase se il segnale di Reset è false, come nel mio caso.
I have to generate a sine waveform with a initial phase but it has to be reentrant.Since I'm simulating a digital down-conversion, I have to multiply the samples of each symbol with an equal number of samples of a sine wave that has a fixed frequency. So, for each symbol I need that sine waveform starts with the phase with which ends the previous sine waveform( related to previous symbol). Also, since I am simulating a 16 QAM modulation,I need to create a sine and a cosine, but the initial phase of 90 ° out of phase with each other. My problem is that SineWaveform ignores phase if reset signal is false, as in my case.

Ciao Paolo,
scusami per il ritardo con cui ti rispondo, ma sono stato abbastanza impegnato. Ho dato un'occhiata al tuo codice, e così com'è strutturato la sine carrier waveform non aggiorna la fase iniziale ad ogni ciclo inquanto è sempre e comunque impostata a phase = 0. Ho implementato un semplice codice che, data una waveform in ingresso, ne ricostruisce la fase e ti restituisce quella dell'ultimo campione della sinusoide in ingresso. A questo punto ti suggerisco di introdurre uno shift register nel for loop. Inizializzandolo a 0 (all'esterno del ciclo), collegandolo all'input phase del Sine waveform generator.vi sei sicuro che ad ogni iterazione la sinusoide portante parte con fase iniziale uguale a quella dell'ultimo campione della sinusoide precedente.
Quindi ricapitolando dovresti:
1) Aggiungere uno shift register al for loop. inizializzato a 0;
2) Collegare l'estremo sinistro dello shift register all'input phase del tuo waveform generator;
3) Inserire il VI allegato a valle del waveform generator;
4) Riportarti l'uscita del Last element phase reconstr.vi all'estremo destro dello shift register precedentemente inserito.
Spero sia tutto abbastanza chiaro, sperando di aver correttamente interpretato le tue problematiche a riguardo.
Saluti
Attachments:
Last element phase reconstr.vi ‏17 KB

New web compression techniques?

Years back I bought Sorenson Squeeze v3. I used it to make .movs and .swfs in multiple sizes. It immediately bought the quality of my online presentations much higher.
About 2 years ago, I purchased FInal Cut Studio and began using h.264 in Compressor instead of .mov's with Squeeze.
I just updated all my applications. FCS2, Flash CS3, etc. I'm wondering now if I can also update my compression techniques.
I looked at On2 VP6 about a year ago, and liked it, but wasnt affected enough to buy.
What's the latest and greatest out there? What are the big boys using now? Is there anything on the horizon that I should be waiting for with Flash CS3 freshly released?
Any insight would be greatly appreciated.
Steven Galvano
Colors Studios

I think the choices are between Flash, and QT H.264... Flash because it's on every computer in the world and is a lot better than the MS solutions, and H.264, because I think it looks the best of all at any given data rate.
Jerry

Peak detection from waveform data

Hi Folks.
This is probably an obvious one but I cant figure it out. In the sine wave detector VI attached the peaks are determined from a simulated signal. I want to be able to do the same with a waveform data (DBL) but it wont let me. How can I extract the signal from the DBL to feed to the rest and make the waveform peak detector VI work right??
Thanks,
Mark.
Solved!
Go to Solution.
Attachments:
Sine Wave Detector 2009.vi ‏140 KB
waveform peak detector.vi ‏40 KB

Use From DDT, found on the Signal Manipulation Palette, to change the Dynamic wire into a Waveform.
Bob Schor

What are the permitted compression techniques for PDF/A-1?

The PDF/A-1 standard does not specify how compression is performed. What are the permitted compression techniques for PDF/A-1?
The information I have gathered (not sure if all are true)
LZW should not be used
JPEG2000 may not be used
Few say, JPEG can be used but its a lossy compression. (PDF standard clearly states lossy compression can't be used)
Also, how to figure out whether a particular file is PDF/A-1a or PDF/A-1b and what kind of compressions are used in that file?
Thanks in advance.

For better or worse, ISO specifications are written in a language from a parallel universe. After dealing with ISO standards for a while, you learn what specific words really mean. I should know - I am chair of the ISO PDF/X task force and co-chair of the PDF/VT task force.
The word should does not specify a requirement. PDF/A does not prohibit lossy compression.
JPEG is always a lossy compression. There is a mode of JPEG2000 compression that is indeed lossless. Note that JPEG and JPEG2000 are totally different compression schemes. (I believe that PDF/A-1 as well as PDF/X-1a and PDF/X-3, based on older versions of the PDF specification do not permit JPEG2000. PDF/A-2 based on ISO32000-1 and PDF/X-4 based on PDF 1.6 do allow for JPEG2000.)
ZIP compression is lossless.
- Dov

How to generate a waveform data table for my data acquisition (GPIB)?

Hi everyone, I'm new to labview. I have a question, how to obtain waveform data(an I/O file so I can plot it in graphic software), rather than just plot a waveform graph which is pretty easy, but I want a waveform data table, how can I do that? Thank you very much!
Message Edited by pack2themoon on 12-12-2008 10:05 PM

Well, I would tell you how to get the data to display on a table however then you will come to another issue.
The issue is that once the data is in a table, you cannot select all the data at once.
You can only select one cell at a time, I'm not sure why, guess it was an oversight of some kind.
There are a few work arounds, and if you are still interested in this aprroach, I will share them with you.
Cory K

File Adapter Data Compression

I'd like to extend file adapter behavior to add data compression features like unzip after read file and zip before write file. I read oracles's file adapter documentation but i didn't find any extension point

if its java mapping, just create a DT with any structure as you wish.
ex.
DT_Dummy
|__ Dummy_field
java mapping does not validate the xml against the DT you created

Data uploading techniques//LSMW

Dear Experts,
As i am very new to SAP HCM. Can any one explain about data uploading techniques in HCM point of view.
ANd what is BDC,PDC
Thanks in Advance
Ram

http://wiki.scn.sap.com/wiki/display/ABAP/Batch+Input+-+BDC
SAP ECC - Plant Data Collection - Time, Attendance and Employee Expenditures (HR-PDC)

Data Migration techniques

Hi Experts,
I want to know about data migration techniques and how we can best use MDM while migrating old version of R/3 to new version of R/3....
I have implemented SAP MDM in cases where we have number of SAP R/3 instances across different region and we were taking data from different R/3 one by one and doing data stan'zation,consolidation, Harmonization and all... I am not talking about all this...
There was a good explanation from Markus Ganser about Duplicate data and identical data... I know all this..but my question is when I have only one SAP R/3 and I still want to implement MDM solution while migrating my old R/3 instance to new one, How can I proceed in this scenario? What is the data migration technique...
I know the common answer will be to use MDM as a middle ware..take Master data from old instance and after consolidation, send it back to new instance and at the same time send tran'data directly to new version... But is this worth doing this? Is there any other approach?
If there is any document on this or any one have idea about data migration techniques while implementing MDM solution than send me documents on [email protected].......
In short, I am looking for below 3 points while doing migration along with SAP MDM
Data Migration techniques
Prerequisites
Methodology in this kind of scenario
Step by step procedure
cheers,
R.n

hi..
here i am sending u the link for complete Data Migration Life Cycle
<a href="http://www.redwoodsystems.co.uk/dataMWhitePaper.html#links">http://www.redwoodsystems.co.uk/dataMWhitePaper.html#links</a>
hope it might be of any use to u
thank you & reward points if useful
Message was edited by:
Dasari Narendra

Waveform data not displaying correctly

Hello,
I am having some trouble displaying some waveform data.
I have a DAQCard-6036E card, and I am bringing in 16 analog signals. The task is set up through MAX, and right now is generic. All I am reading is random noise in an attempt to see if my VI even works. The acquisition mode is set to continuous, with a rate of 1k, and number of samples set to 100.
The problem is that I get very intermittent data on the front panel indicators. Only several of the indicators work, and its choppy at best. What am I doing wrong here? Is it a problem with my acquisition mode?
Thanks,
Alex
Attachments:
DAQ (CompactDAQ).vi ‏39 KB
DAQ.vi ‏137 KB
DAQ (Air Data Computer).vi ‏35 KB

Hi Alex,
I just tried the your VI with the while loops deleted out of the
subVI's and as far as I could see the indicators in the first two
columns of the front panel were updated as expected. The other
indicators are not wired so they will not update. It might look like
some of the wired indicators are not updating each time if the value
doesn't change - ie the potential on the line is the same as the last
read. You might try putting a known signal on each of your test lines
instead of using noise to test your system to verify that the voltage
is changing each time.
If you are still having trouble, post again. I've attached a copy of the VI I used for test.
Regards,
Micaela N
National Instruments
Attachments:
DAQ.zip ‏95 KB

How can i add formula to waveform data type?

I want to interpolate my acquired data in terms of voltage to temperature. For this i need to use polynomial equation, but i am not able to directly attach formula node to the waveform data type. I want to know how can i add formula node to waveform data, and then also have it real time.

One additional thought that may be helpful. Once you get the Y component of the waveform graph you are going to attempt to wire this into a formula node. You can do one of two things with that array. Put a for loop around the formula node and index the array into and out of the for loop. This allows you to do scalar math on the data. It is also possible to index and array in a formula node. The following KnoledgeBase demonstrates how to do this: http://digital.ni.com/public.nsf/3efedde4322fef19862567740067f3cc/9d72b6069346942386256a0d00604ed4?OpenDocument

Using Data Compression on Microsoft SQL 2008 R2

We have a very large database which keeps growing and growing. This has made our upgrade process extremely troublesome because the upgrade wizard seems to require close to 3 times the database size of free space to even start.
As such, we are considering activating DATA COMPRESSION on the PAGE level in Microsoft SQL Server 2008 R2. This technology is native to the SQL Server and compresses the rows and the pages so that they do not take up more space than necessary.
Traditionally each row take up the space of the maximum of all the fields even though only part of a field is filled with data.
[Blog about Data Compression|http://blogs.msdn.com/b/sqlserverstorageengine/archive/2007/11/12/types-of-data-compression-in-sql-server-2008.aspx]
Our idea is to use this initially on the Axxx tables (historic data) to minimize the space they take by using for example:
ALTER TABLE [dbo].[ADO1] REBUILD PARTITION = ALL
WITH (DATA_COMPRESSION = PAGE)
On a test database we have seen tables go from 6GB of space to around 1,5GB which is a significant saving.
MY QUESTION: Is this allowed to do from SAP point of view? The technology is completely transparent but it does involve a rebuild of the table as demonstrated above.
Thanks.
Best regards,
Mike

We are using Simple recovery model, so our log files are pretty small.
Our database itself is about 140GB now and it keeps growing.
We've also reduced the history size to about 10 versions.
Still, some of our tables are 6-10GB.
Some of the advantages of data compression is to also that it improves disk I/O at the cost of slightly higher CPU, which we are pretty sure our server can handle.
Mike

Sine waveform data Compression Techniques

Similar Messages

Maybe you are looking for