How to get data of more than 100 fields by Bulk API V2.0?

Question

Hi all,
I'm trying to get data from Eloqua by Bulk APIs because of big data.
But my Contact has 186 fields (more than the Bulk export limitation 100). I think I need to get all data by 2 exports.
How could I match 2 parts of a row and join them together?
I'm afraid that any edit of data between the 2 syncs for the 2 exports would make the order different.
E.G.:
    1. any record is deleted or edited(make it not matching the filter) after getting data of the first part and before getting the second part, then every one behind it would move up in the second part result.
    2. data of some fields (included in both parts) are changed between the 2 syncs, then values of the second part are newest but values of the first part are old.
Any suggestions would be expected.
Thanks,
Biao

Answer 1

bhuang -
I don't know that you're ever going to get around the fact that things will change in your database while you're syncing the data. You're going to have to have some way to create exceptions on the sync side.
If I was pushing data from Eloqua to another database and had to deal with the problem of matches changing while I was syncing, I'd create a few extra columns in my database to keep track of sync status for that record. Or create another small table to track the data mapping. Here's how I'd do it.
I'd have two extra columns:  'mapped fields 1' and 'mapped fields 2'. They'd both be datetime fields.
I would only do one set of syncs at a time. First, sync every record for the email + 99 fields. Do the entire list. For each batch, set the datetime of the batch in 'mapped fields 1' column.
I'd then sync all records for email + other 86 fields. Do the entire list again. For this batch, set the datetime of each batch in their 'mapped fields 2' column to now().
For any records that had only 'mapped fields 1' filled, but 'mapped fields 2' was empty, I'd re-run the second Eloqua API query using the email as the lookup value. If no results were returned, I'd delete the row. Otherwise, update and set 'mapped fields 2' to now
For any records that had only 'mapped fields 2', I'd re-run the emails against the first Eloqua API query, fill in the missing data, and set 'mapped fields 1' to the current datetime. If the record wasn't returned, delete the row as it's probably not in the search any longer.
Finally, set 'mapped fields 1' and 'mapped fields 2' to empty for all records, since you know the data is in sync. This will let you use the same logic above on your next sync.
Does that make sense? It's not super clean, but it'll get the job done, unless your syncs are taking a ridiculous amount of time and your data changes super often.