CS4 Crashes when attempting to change color of characters in story

Question

Below is a function that we have been using since forever. It changes the color of a specified number of characters in a story starting at a specific position in the textmodel.<br /><br />The crash happens when processing the command (immediately after "11" is displayed).<br /><br />classID is kTextAttrColorBoss.<br /><br />Any help would be greatly appreciated.<br /><br />ErrorCode ApplyTextAttributes(ITextModel * pTextModel, int32 position, int32 length, UID swatchUID, ClassID classID)<br />{<br />     ErrorCode rc = kFailure;<br />     do<br />     {<br />          InterfacePtr<ITextAttrUID> textAttrUID_Color(::CreateObject2<ITextAttrUID>(classID));<br />          if (textAttrUID_Color == nil)<br />          {<br />//               ASSERT_FAIL("TSCore::CreateCharStyle: color invalid");<br />//               CAlert::ErrorAlert("textAttrUID obj is nuil");<br />               break;<br />          }<br /><br />          if (swatchUID == kInvalidUID)<br />          {<br />               ASSERT_FAIL("TSCore::CreateCharStyle: ResolveRGBColorSwatch failed");<br />//               CAlert::ErrorAlert("Invalid color UID");<br />               break;<br />          }<br />          <br />          // Set our color attribute and store it:<br />          textAttrUID_Color->SetUIDData(swatchUID);<br />          <br />          InterfacePtr<ICommand> pApplyTextAttrCmd ( Utils<ITextAttrUtils>()->BuildApplyTextAttrCmd(pTextModel, position,  length, textAttrUID_Color, kCharAttrStrandBoss) );<br />          //pApplyTextAttrCmd->SetUndoability(ICommand::kUndoNotRequired);     <br />CAlert::WarningAlert("11");<br />          rc = CmdUtils::ProcessCommand(pApplyTextAttrCmd);<br />CAlert::WarningAlert("12");<br />          if (rc != kSuccess)<br />               CAlert::WarningAlert("Error on ApplyTextAttrCmd");<br />     }while(false);<br />     <br />     return rc;               <br />}

Answer 1

Hi Rick,
I've just been chasing a crash in the Debug version of CS4 InDsign which was asserting with Protective shutdown.
Like you the manifestation was on a call to ProcessCommand.
I don't know if it's the same, but on the off chance I'll detail what I discovered.
I traced back the cause of the crash to an earlier call through our ASB-CSB using the function template and macros, Process and make_functor.
The function template Process which is in the SelectionASBTemplates.tpp sets up a abortable command sequence on the stack.
If the result from the CSB function wasn't kSuccess then the sequence was aborted.
Unfortunately in our CS2 version of the ASB/CSB's we were rather naughtily using the return to pass  a value and not an ErrorCode.
the result was that our call was forcing an abort everytime.
I've duplicated and renamed the Process function at the top of our ASB code and missed out the command sequence and it works fine.
This isn't really the best solution and I'll have to investigate further but at least it solves the crash.
Below is the new version of Process. I hope it helps you (or someone else out).
All the best
Caerwyn  Pearce
#if (CREATIVE_SUITE_VERSION == CS2)
// Use standard function template from SelectionASBTemplates.tpp for CS2.
// It doesn't seem to cause problems, and I'd prefer not to fix it if it's working.
#define ATLASB_Process        Process
#else
//    DESCR:        Call each CSB suite to process the request, wrapping the entire call in a sequence.
//                Compare this with the one in SelectionASBTemplates.tpp.   
//                Essentially I've removed the command sequence, and the possibility of an abort.
//                For some reason this was causing a Protective shutdown in CS4.
//                Also note the abort takes place if the return value from Process is not kSuccess(aka 0).
//                For GetSelection we are returning ATLCSB::SelectionType which has a value of -1, -2 or -3
//                which is most definitely not kSuccess and which was provoking an abort.
//                However that's not the end of the problem, and the line ErrorUtils::PMSetGlobalErrorCode
//                seems to be causing problems else where outside of the Process function template.
//                It may be that our code is not resilient to aborting command sequences.
//    NOTE:        Most definitely doesn't abort the sequence regardless of errors.
//    RETURNS:    Error Code.
template <typename Functor>
typename Functor::result_type ATLASB_Process (Functor& f, typename Functor::argument_type object, PMIID iid)
    InterfacePtr<const IIntegratorTarget>                     iIntegratorTarget (object, UseDefaultIID ());
    std::auto_ptr<IIntegratorTarget::TargetSuiteCollection> selectionSuites (iIntegratorTarget->GetTarget (iid));
    ErrorCode                                                errorCode = kSuccess;
    if (!selectionSuites->empty())
        for (int32 i = selectionSuites->size () - 1; i >= 0 && errorCode == kSuccess; i--)
            typename Functor::argument_type        iUnknown = static_cast<typename Functor::argument_type>(static_cast<IPMUnknown*>((*selectionSuites) [i]));
            errorCode = f(iUnknown);
    return (errorCode);
template <typename Functor>
typename Functor::result_type ATLASB_Process (Functor& f, typename Functor::argument_type object)
    return (ATLASB_Process (f, object, GetDefaultIID (object)));
#endif