Using powershell to check and removing duplicate entries

Hi Guys,
I have been trying to get my head around powershell to check a csv file. It needs to read in each line in the csv and compare it to the last one and only pull out the entry with the newest time stamp. The data is something like this (btw it has no headers
pulled into the csv file).
UID,Type,Sub,Blank,deviceID,Brand,V#,OS,DType,Blank,Date/time
TT1234567,PRINT,7DAY_SUB,,7DD758034,iPad3,6,IOS,Tablet,,2014-09-22T10:59:40.345Z
TT1234567,PRINT,7DAY_SUB,,7DD758034,iPad3,6,IOS,Tablet,,2014-09-22T11:00:40.500Z
TT1234567,PRINT,7DAY_SUB,,f04f7c213b0f,kftt,2,KINDLE,Tablet,,2014-09-23T11:25:16.303Z
TT1234567,PRINT,7DAY_SUB,,f04f7c213b0f,kftt,2,KINDLE,Tablet,,2014-09-23T11:25:18.303Z
TT1234567,PRINT,7DAY_SUB,,08606eb42491,nexus7,4.4,ANDROID,Tablet,,2014-09-24T14:48:23.668Z
TT1234567,PRINT,7DAY_SUB,,08606eb42491,nexus7,4,4,ANDROID,Tablet,,2014-09-24T14:48:24.700Z
[email protected],DIGITAL,DIG_SUB,,08606eb42491,nexus7,4.4,ANDROID,Tablet,,2014-09-24T15:09:08.359Z
[email protected],DIGITAL,DIG_SUB,,08606eb42491,nexus7,4.4,ANDROID,Tablet,,2014-09-24T15:09:09.468Z
TT1234567,PRINT,7DAY_SUB,,7DD758034,iPad3,6,IOS,Tablet,,2014-09-25T11:59:40.345Z
TT1234567,PRINT,7DAY_SUB,,7DD758034,iPad3,6,IOS,Tablet,,2014-09-25T11:59:45.345Z
Output to file should be like the following.
TT1234567,PRINT,7DAY_SUB,,7DD758034,iPad3,6,IOS,Tablet,,2014-09-25T11:59:45.345Z
TT1234567,PRINT,7DAY_SUB,,f04f7c213b0f,kftt,2,KINDLE,Tablet,,2014-09-23T11:25:18.303Z
TT1234567,PRINT,7DAY_SUB,,08606eb42491,nexus7,4,4,ANDROID,Tablet,,2014-09-24T14:48:24.700Z
[email protected],DIGITAL,DIG_SUB,,08606eb42491,nexus7,4.4,ANDROID,Tablet,,2014-09-24T15:09:09.468Z
Same UID can have multiple devices we want to know all the devices and the latest time stamp. As you can see that
TT1234567 used the ipad on the 2014-09-22T11:00:40.500Z
but then used it again on the 2014-09-25T11:59:45.345Z we only want the newest time stamp for that device.
I hope you guys can help me out.

Hi Nguyen,
First of all, add a header for your csv (simplifies things a bit):
ID,Type,Sub,Blank1,deviceID,Brand,VNum,OS,DType,Blank2,DateTime
Then this single line will get you what you want:
import-csv test.csv | group deviceID | %{$_.Group | Sort DateTime | Select -Last 1}
Please note:
Make sure your Csv conforms to Csv Standards (especially that the Delimiter is
only used as delimiter, which in your sample data was not the case). If you cannot ensure this, you need to create a validation routine and fix it before you use this line. Too many delimiters will drop the DateTime from the list, leaving a blank datetime,
causing the filtering to fail.
Cheers,
Fred
There's no place like 127.0.0.1

Similar Messages

HT2905 No Display Duplicates under File. How to find and remove duplicate items in your iTunes library

I now have iTunes ver 11.0.4.4 under Windows 7. I lost all iTunes stuff when updating to Windows 7. I have loaded thousands amd thousands of music files from backup disks, but there are many duplicates. I am attemping to re-establish my old library. I used to be able to remove duplicates quickly with the old iTunes. The new iTunes doesn't seem to offer the same service. Is there any way to remove duplicates quickly, or must I do it one by one?

When deduping use Shift > View > Show Exact Duplicate Items as this is normally a more useful selection. You need to manually select all but one of each group to remove. Sorting the list by Date Added may make it easier to select the appropriate tracks. If you have multiple entries in iTunes connected to the same file on the hard drive then don't send to the recycle bin. Use my DeDuper script if you're not sure, don't want to do it by hand, or want to preserve ratings, play counts and playlist membership. See this thread for background and please take note of the warning to backup your library before deduping.
(If you don't see the menu bar press ALT to show it temporarily or CTRL+B to keep it displayed)
See also HT2905: How to find and remove duplicate items in your iTunes library
tt2

Union all-distinct and remove duplicates from nested table?

Hi all,
I need a select that will bulk collect some data in my nested table.
I have two tables from which I need to select all the accounts only once.(remove duplicates).
Tried to search on the forum...but no luck.
I have a table with one column:
create table a1(account_no number);
and a second table with 3 columns.
create table a2 (account_no number, name number, desc varchar2 (100));
I have a nested table like:
table of a2%rowtype;
Can I select from this two table in one select and put in my nested table just one row per account?
if a I have in a 2a row like :
1 'test' 'test2'
2 aaaa aa
and in a1 a row like:
1
I want to put in my nested table just (1, null,null and 2,aaaa, aa)) or (1,test,test2 and 2,aaaa, aa). it does no matter what row from those two I insert.
Second question:
If I use:
BANNER
Oracle9i Release 9.2.0.5.0 - Production
PL/SQL Release 9.2.0.5.0 - Production
CORE     9.2.0.6.0     Production
TNS for 32-bit Windows: Version 9.2.0.5.0 - Production
NLSRTL Version 9.2.0.5.0 - Production
SQL>
what is the best solution to remove duplicates from a neste table like mine?
I thought that I can build another nested table and loop in my first nt and for each row I check in there was the same account in previous lines.
it will be like:
for i in 1....nt_first.count loop
for j in 1..i loop
    --check if my line was in previous lines. if it was...do not move it in my second collection
end loop;it is this best option in oracle 9i?

I have a table with one column:
create table a1(account_no number);
and a second table with 3 columns.
create table a2 (account_no number, name number, desc varchar2 (100));
all I need are the accounts. the rest ar extra data
that I can ignore in this step. But if it is
available, it is ok to use it.
using one select in this case is is much better that
trying to remove duplicates parsing some nested table
with FOR many times?
Thankshi,
try to use union. Union automatically removes duplicates between two or more tables.
with t1 AS
       (select '3300000' account_no FROM DUAL UNION
        select '6500000' account_no FROM DUAL union
        select '6500000' account_no FROM DUAL union
        select '6500000' account_no FROM DUAL union
        select '6500000' account_no FROM DUAL
       select * from t1ACCOUNT_NO
3300000
6500000

HT2905 how do i find and remove duplicate songs in itunes 11.1.3?

i transferred my itunes to a new computer today, and somehow half of my library was duplicated. no i cant seem to find a way to get rid of the duplicates without selecting them one at a time. any ideas?

Apple's official advice on duplicates is here... HT2905: How to find and remove duplicate items in your iTunes library. It is a manual process and the article fails to explain some of the potential pitfalls.
Use Shift > View > Show Exact Duplicate Items to display duplicates as this is normally a more useful selection. You need to manually select all but one of each group to remove. Sorting the list by Date Added may make it easier to select the appropriate tracks, however this works best when performed immediately after the dupes have been created. If you have multiple entries in iTunes connected to the same file on the hard drive then don't send to the recycle bin.
Use my DeDuper script if you're not sure, don't want to do it by hand, or want to preserve ratings, play counts and playlist membership. See this thread for background and please take note of the warning to backup your library before deduping.
(If you don't see the menu bar press ALT to show it temporarily or CTRL+B to keep it displayed)
Alternativey see this migrate iTunes library post, you may want to revisit the transfer method.
tt2

Help required on mapping to remove duplicate entries

Hi all experts,
I need some help in mapping. I am new to this tool. How would I remove duplicate entries in source payload? My structure is as follows
Message Structure
<List> 1-1
<ParentNode> 0-unbounded
<Child1> 0-1
<Child2> 0-1
Source:
<List>
     <ParentNode>
          <Child1>PK1</Child1>
          <Child2>SomeValue1</Child2>
     </ParentNode>
     <ParentNode>
          <Child1>PK1</Child1>
          <Child2>SomeValue1</Child2>
     </ParentNode>
     <ParentNode>
          <Child1>PK2</Child1>
          <Child2>SomeValue2</Child2>
     </ParentNode>
</List>
Target I would like:
<List>
     <ParentNode>
          <Child1>PK1</Child1>
          <Child2>SomeValue1</Child2>
     </ParentNode>
     <ParentNode>
          <Child1>PK2</Child1>
          <Child2>SomeValue2</Child2>
     </ParentNode>
</List>
Any help?
Thanks so much all

Hi,
Please note that in my reply I asked you to map ParentNode with <Child1> Please see the below.
<Child1>-->RemoveContext Node function>Split by Value(set the property as on value change)->CollapseContext---<ParentNode>
Secondly: introduce split by value(for each value ) after collapse context for child1 and child2. as shown below.
<Child1>-->RemoveContext Node function>Split by Value(set the property as on value change)->CollapseContext>Split by value(for each value)----</Child1>
<Child2>-->RemoveContext Node function>Split by Value(set the property as on value change)->CollapseContext>Split by value(for each value)----</Child2>
it will surely work,

Remove duplicate entries from dropdownlist in web dynpro abap

How to remove duplicate entries from dropdownlist in web dynpro abap? Can someone please help me
I have maintained the data in the z table wherein the records of particular fields are repeated but when i show that record in the Web Dynpro application dropdown list, the user should only be able to view the unique data for selection of that particular field.

Hi,
try this code in init method.
use the
Delete adjacent duplicates.
<set the table>
select <f1> from <table> into TABLE <Itab> whre <condition>.
DELETE ADJACENT DUPLICATES FROM <Itab> COMPARING <f1>.
lo_nd_vbap->bind_table( new_items = <itab> set_initial_elements = abap_true ).

Referencing multiple cells and removing duplicate values.

Hello.
I have a very complicated question, to be honest I am not totally sure if numbers is capable of handling all of this but it's worth a shot.
I am working on a spreadsheet for organising a film. I've had the templates for years but I'm now using numbers to automate it as much as possible. Everything was going well until I hit the schedule/callsheet.
On other sheets I can tell it to "look up scene two" it will then look up the correct row and find everything I need. On the callsheet however I might say "we're filming scenes two, five and nine" and numbers gets confused with the multiple values, Is there anyway around this?
Also, if there is, I have a more complex question to ask. Is it possible for numbers to find and remove duplicate data? For example lets say scene two and five require Alice, but scene nine requires bob. If numbers just adds that info together it will display the result "Alice Alice Bob", is there a way to get it to parse the text, recognise the duplicate value and remove the unnecessary Alice?
I realise that numbers has limitations so it may not be able to do everything I want, however every bit I can automate saves me hours so even if I can only get half way there, totally worth it.
Thanks in advance for any help you can offer, all the best.

Ah excellent thank you.
I've modified it to there are now multiple (well only four for now until I get this in better shape) indexes for finding a scene. And assigning each block to a new row.
I only have one slight reservation about this. If I create 10 rows, it totally works, most of the time we'll only shoot three scenes a day so it's just blank space... However Murphy's law will inevitable raise its ugly head and put me in a situation where we are shooting 11 scenes in a day.
For countif, I think I get what you mean... Kinda. Basicially I would create a cell which combines the character strings from each scene into one long scene. Then I would have 100 extra cells (Lets say 100 so I'll never run out) each linked to the cast list, using the character name as a variable. These cells will each parse through the string to find their character name. If it appears then a true value comes up. This will remove duplicates as each cell will only respond once. So if Alice appears in every scene shooting that day, the cell will still only light up once. Is that it.
One other question. Whenever I combine filled cells with blank cells. I usually gets the data from the filled cells, with a 0 for each blank cell. Usually this isn't a problem, but if I want to keep this flexible then I'll have quite a few blanks. The actor example above could result in 98 zeroes. Is there anyway to have blanks just not show up at all.
I'll email the spreadsheet in a moment, a lot of it is still rough and under construction, but you should be able to see where it's all going hopefully.
Thanks again, you have been extraordinarily helpful.

How can I find and remove duplicate photos in iPhoto?

What is the best way to find and remove duplicate photos in iPhoto?

Are you seeing these duplicates in iPhoto or via the Finder? If it's in the iPhoto window then you can use one of these applications to identify and remove duplicate photos from an iPhoto Library:
iPhoto Library Manager - $29.95
Duplicate Cleaner for iPhoto - free
Duplicate Annihilator - $7.95 - only app able to detect duplicate thumbnail files or faces files when one library has been imported into another with iPhoto 8 and earlier.
PhotoSweeper - $9.95 - This app can search by comparing the image's bitmaps or histograms thus finding duplicates with different file names and dates.
I also prefer iPLM as it is more than just a dup finder. It's a the most versatile iPhoto utility available.
OT

Find and remove duplicates advice?

hi all,
would it be possible to get a little advice and some feedback on others' experience with finding and removing duplicates from the aperture database?
i am doing this now for scans, pdf's, images and other data on my mac book pro and on my mac pro and it would be helpful for me to know what is possible for me to get done with the aperture database.
TIA

I haven't used it myself but Duplicate Annihilator it is mentioned here often as a solution for this.

TS4062 how do i remove duplicate entries from my phone directory?

question; how do i remove duplicate entries from my phone directory?

go to your contacts icon and choose the one you want to delate
Choose edit, top right and go (scroll) to the bottom of the page and press delete contact

How do I find and remove duplicates on the 11.2 itunes?

How do I find and remove duplicates on the 11.2 itunes?

duplicate annihaitor
But often duplicates are a symptom and you really need to address the cause - more information is needed to help out with that
LN

How to streamline and remove duplicate files on my HD before I backup?

How to streamline and remove duplicate files on my HD before I backup?
is there a good program or freeware that can automate and isolate when I have
duplicate files hiding in the many folders on my Drive... Yes, in a perfect world
I should know better than having a Drive that is not perfectly organized
I think there is something called SpringCleaning for windows
is there something like this for Mac?
Time machine won't let you remove something after you realized you backed
up two of the same files... Thanks for ideas

+is there a good program or freeware that can automate and isolate when I have
duplicate files hiding in the many folders on my Drive.+
You should back up your drive with Time Machine before you delete any files in case you make a mistake.
Tidy Up! - 1.3.2
http://www.versiontracker.com/dyn/moreinfo/macosx/17908
+Time machine won't let you remove something after you realized you backed
up two of the same files.+
Click on the Time Machine browser icon on the dock. Locate the duplicate file you wish to delete and highlight it. Click on the button with the gear on and choose Delete "all backups of".

How can I find and remove duplicate photos from my computer?

How can I find and remove duplicate photos from my computer?

Terence,
Yes, the duplicates appear in the iPhoto window. I have folders with same name occuring two or even three times sometimes with exactly the same set of photos (ie photos with the same ID) and sometimes with a limited set of photos. Other folders appear only once.
I normally take my MacBook with me if I'm shooting a lot of photos, such as on holiday, and then want to transfer them to my iMac when i get home. Other times I transfer the camera's memory stick directly to my iMac.
I have great difficulty transferring the photos from my MacBook to my iMac. I don't want to store my photos on my iDisk due to the length of time it takes to upload them.
Thanks again.
Simon

How can I identify and remove duplicate file (audio & photo)

How can I identify and remove duplicate file (audio & photo)? I have quite a few libraries created as backup files.

Terence,
Yes, the duplicates appear in the iPhoto window. I have folders with same name occuring two or even three times sometimes with exactly the same set of photos (ie photos with the same ID) and sometimes with a limited set of photos. Other folders appear only once.
I normally take my MacBook with me if I'm shooting a lot of photos, such as on holiday, and then want to transfer them to my iMac when i get home. Other times I transfer the camera's memory stick directly to my iMac.
I have great difficulty transferring the photos from my MacBook to my iMac. I don't want to store my photos on my iDisk due to the length of time it takes to upload them.
Thanks again.
Simon

How to identify and remove duplicate plugin effects from AE cs5 please?

hello,
i installed some trial plug in effects for AE and PPRO cs5
cycore, boris, trapcode
the PPRo went great
however, with the AE i get notices that multiple instances of 'such and such' effect
is installed more than one time...there's about 15 that give me a warning each time i start AE...
i tried my best to locate any duplicates but no luck
How do you identify and remove duplicate plug in effects for AE CS5?
thanks,
j

hello,
it is nice when you can answer your own question...
looked in 'common...media core...' and found the new blue trial plugins
also had the duplicates of the new blue plugins in the AE plugins 'effects' folder
deleted the plugins from the 'media core' folder
no more warning signs...
cool deal
back to work,
j

Using powershell to check and removing duplicate entries

Similar Messages

Maybe you are looking for