How do I filter a column by specific values?

Question

Hello,
I have imported a large .csv log file that contains many different columns and values into DIAdem. One of the columns contains a hexadecimal value field, and I need to filter the column by the last byte of this hex field. For instance, one of the values might be "CFFD0F3" and I need to find all rows that have "F3" as the last byte. I also need all of the other columns to be filtered in this same way.
There doesn't appear to be any easy way to do this in DIAdem, and I've searched online and through the discussion forums for ways to accomplish it. Whenever I search for the word "filter" it always leads me to FFTs or highpass/lowpass filters. I just want to filter a column like Excel does, but my log file contains around half a million entries, so I can't open it entirely in Excel.
I've attached a screenshot of what some of my columns look like, but for privacy reasons, I can't attach the log file.
Does anyone have any ideas?
Thanks!
Solved!
Go to Solution.
Attachments:
screenshot.png ‏159 KB

Answer 1

Hi Ben,
I cooked up a nice little application for you.  Since your post clearly stated that you were dealing with millions of data values, I took pains to implement this application as efficiently as possible and to include status dialogs to let you know what the script is doing at all times and which actions take the most time, in case we need to try to optimize it further for your purposes.  You will need to extract the 3 files in the attached ZIP file to the same folder somewhere.  Then you need to run the "ExtractIDGroups Add.VBS" script to add the VIEW context menu and register the UserCommands the VIEW context menu will be calling.  We can set up your DIAdem to always run this VBScript whenever you launch it, once you're happy with the result.
You will use the new VIEW context menu in the following way.  Load your data file into DIAdem, then show those loaded data channel in a VIEW table, just like your posted screenshot.  Now select and right-click on the "Arb ID" channel (or whichever channel has the Hex addresses) and choose the new "Extract ID Groups" context menu.  A user dialog will pop up on the left of your computer screen listing all the enumerated final 2 Hex suffix characters in that channel you right-clicked on.  Select one or more of the desired Hex suffixes (shift or control key for multiselect), then click on the <OK> button.  You will then see each of those selections broken out as a new Group in your Data Portal.  You can select the "Extract I Groups" multiple times if you decide to go back and add a few more ID Groups.
The application is currently set up to sort the raw data in-place by the "Hex Suffix" values and leave them that way-- this along with leaving the "Hex Suffix" channel in the raw data Group makes it faster to repeat ID Group extraction from the same raw data set.  This is also the more efficient way of extracting multiple ID Groups, which I'm guessing you will want to do.  Each extracted ID Group is sorted at the very end by the values of first channel in that Group.  So if you want to sort those extracted ID Groups by the "Abs Time" or "Rel Time" channel, say, you need to drag that sort-channel in the Data Portal from its start position to the very first channel position in that raw data group BEFORE selecting the "Extract ID Groups" context menu.
Let me know how this works for you,
Brad Turpin
DIAdem Product Support Engieer
National Instruments
Attachments:
ExtractIDGroups.zip ‏4 KB