Lightroom processes for ages after withProlongedWriteAccessDo

Question

Hi,
I am considering a switch from Expression Media to Lightroom, and I wrote for that a Lightroom plugin that imports all my metadata from an EM XML file. Functionaly, it works great.
However, I have a performance issue problem: if I run the plugin on my whole photo collection (>30.000 items), then Lightroom gets crazy with the CPU and the memory after once my code returns.
The processing time seems exponential. If I run the plugin with 1.000 items, the return is almost instantaneous. On 5.000 items, it takes several minutes. With 30.000 items, it still wasn't finished after 30min, I killed Lightroom.
My guess is that Lightroom does some kind of post-processing, or maybe that's because Lightroom prepares the "undo".
Is that a known problem ?
Here is the skeleton of my code. Once doStuff2 returns, Lightroom does who knows what, and then doStuff returns too. This time seems exponential with the number of database changes my code causes.
function doStuff2(context, progress)
     local catalog = LrApplication.activeCatalog()
     local photos = catalog:getTargetPhotos()
     parseXml()
     for completed, photo in ipairs( photos ) do
          if progress:isCanceled() then return end
          photo:addKeyword(...)
          setMetadata(...)
end
function doStuff()
     local catalog = LrApplication.activeCatalog()
     catalog:withProlongedWriteAccessDo(
               title="Import from Expression Media",
               func=doStuff2,
               caption="Initializing plugin",
               pluginName="Expression Media Import",
               optionalMessage = "The plugin will first read your Expression Media XML file and then import the data into Lightroom for "
                    .. ( # catalog:getTargetPhotos() ) .. " photos"
end
import 'LrTasks'.startAsyncTask( doStuff )

Answer 1

Very possibly.
I've been really stoked with my new multi-phase catalog accessor - strongly recommend writing something like it for yourselves (see source code in other thread, or contact me for fresh stuff).
Here's an example from my most recent plugin:
local s, m = cat:update( 10, "Assure Folder Collection Set", function( context, phase ) -- 10 is seconds to get in.
    -- function can take parameters, but not used here.
    if phase == 1 then
        set = catalog:createCollectionSet( folder:getName(), parent, true )
        return false -- keep going
    elseif phase == 2 then
        leafPhotos = folder:getPhotos(false)
        if #leafPhotos > 0 then
            return false
        else
            return true -- done
        end
    elseif phase == 3 then
        coll = catalog:createCollection( str:fmt( '[^1]', folder:getName() ), set, true )
        app:logVerbose( "Created/assured \"Folder Leaf Photos\" collection ^1 in '^2'", folder:getName(), set:getName() )
        return false
    elseif phase == 4 then
        local collPhotos = coll:getPhotos()
        if #collPhotos > 0 then
            local collPhotoSet = tab:createSet( collPhotos )
            for i, photo in ipairs( leafPhotos ) do
                if not collPhotoSet[photo] then
                    toAdd[#toAdd + 1] = photo
                end
            end
        else
            toAdd = leafPhotos
        end
        if #toAdd > 0 then
            coll:addPhotos( toAdd )
        end
        return true
    end
end )
Note: each phase is a new transaction.
In your case Oliver:
local keyword
local children
local s, m = cat:update( 10, "Keyword business", function( context, phase )
     if phase == 1 then
          keyword = catalog:createKeyword( "Test", {}, true, nil )
          return false -- continue to 2nd phase.
    elseif phase == 2 then
          children = keyword:getChildren()
          -- do something to catalog with children
          return true -- or nil.
     end
end )
This would not make sense if nothing further involved catalog writes (in which case just exiting a simple single-phase with-do method would suffice), but you get the idea, no?
R