Home for HMNL Enterprise Computing

Enterprise Computing

A Tale of Two Responses

Ian Tree  19 July 2006 11:24:27

A Tale of Two Responses


  • Introduction
  • The C-API Documentation
  • LotusScript Uncovered
  • LotusScript Gotcha #1
  • LotusScript Uncovered Second Attempt
  • The C-API Solution
  • LotusScript Gotcha #2
  • The Java Story

Introduction


Just the other day I set about the task of "reviving" an old C-API application that hadn't seen the light of day for some four or five years. It was a matter of removing layers of accumulated dust and doing a quick "due diligence" review of the code before doing a little regression testing to ensure that all was well. It was during the quick code review that something caught my eye and led to the (to my mind) interesting saga that I now recount below.

The piece of code that caught my eye was in a part of the application that was processing response documents, the function was passed two document handles and needed to compare the response collections under each main document and do some processing according to the state of the collections. The function had been implemented using two views and combination of NAVIGATE_NEXT_PEER and NAVIGATE_NEXT_CHILD reads to effectively "walk" the response collections in the views. I was quite sure that I had implemented response document access in C-API applications before using a much more elegant and efficient approach, I just couldn't remember how I had done it. My first action was to check through my C-API code snippet database to see if I could locate a sample, no luck! I then tried some of my project archives and again came up empty handed. There was no option left, there was only one thing I could do, so, making sure that my SpyWare filters were set to maximum so that no-one could possibly see what I was doing I consulted the C-API DOCUMENTATION !

The C-API Documentation


Starting with the User Guide (6.5) I soon located the section that explicitly deals with response documents (11 Miscellaneous Topics / 3 Response Documents), within that venerable document there is a section on "Reading Response Documents", there I read the following.

Hierarchical views display response documents indented below the corresponding main document.

Since a COLLECTION object, obtained by NIFOpenCollection, preserves the ordering and indenting of the view, the best way to find response documents in a database is to use a hierarchical view COLLECTION and NIFReadEntries.

A response reference list links a child document to its parent. However, the parent document does not contain a link to the child. Given a parent document, then, the only practical way to find the corresponding child documents is to navigate to the child using a hierarchical view.

The section described exactly the technique that I had found in the function in my application and notice the code that I have highlighted above, it is the only practical way to do it! I was disappointed but not yet defeated, I still had some kind of trace memory that said that there was a better way to do this. Undaunted I turned to the C-API Reference Manual (6.5) and searched in the function view for "Response Document", 1 hit on the NSFItemDeleteByBLOCKID function where the term is used in a comment in the code sample, that was no help. I then searched for "Response" and there were 24 hits, which seemed more hopeful. Unfortunately none of the documents threw any light on an alternate technique.

But, hang on a second, all of the high level APIs LotusScript/COM, Java and C++ API, provide "convenience" methods to access response documents as collections, I cannot believe that they implement these methods using the "only practical way" as documented in the C-API documentation. Maybe a little digging.......

LotusScript Uncovered


I coded up a tiny LotusScript agent with the intention of running it aginst a server database that contained some documents with response documents and look at the CLIENT_CLOCK trace to see how it was interacting with the server. The code is below in full.

   Dim sessCurrent As New NotesSession
   Dim dbTarget As NotesDatabase
   Dim dcAll As NotesDocumentCollection
   Dim docTarget As NotesDocument
   Dim docResponse As NotesDocument
   Dim iDocs As Integer
   Dim iResps As Integer
   Dim iIndex As Integer
   Set dbTarget = New NotesDatabase("Soho1/Server/DeltaPhi", "HMI/Dev/ResLib.nsf")
   If Not dbTarget.IsOpen Then
       Print "ERROR: Failed to open target database."
       Exit Sub
   End If
   Set dcAll = dbTarget.AllDocuments
   If dcAll Is Nothing Then
       Print "ERROR: Could not get the All Docs collection"
       Exit Sub
   End If
   If dcAll.Count = 0 Then
       Print "ERROR: All docs collection is empty."
       Exit Sub
   End If
   Set docTarget = dcAll.GetFirstDocument
   While Not docTarget Is Nothing
       If Not docTarget.IsResponse Then
           iDocs = iDocs + 1
           Set docResponse = docTarget.Responses.GetFirstDocument
           While Not docResponse Is Nothing
               iResps = iResps + 1
               Set docResponse = docTarget.Responses.GetNextDocument(docResponse)
           Wend
       End If
       Set docTarget = dcAll.GetNextDocument(docTarget)
   Wend
   Print "INFO: Scan completed, " + Cstr(iDocs) + " documents read and " + Cstr(iResps) + " responses read."


Running the agent resulted in a LotusScript error "Document is not from this collection" being thrown by the following line Set docResponse = docTarget.Responses.GetNextDocument(docResponse). Ahhhhh! This triggered another trace memory about a little known and documented Gotcha in LotusScript.

LotusScript Gotcha #1


LotusScript has a safety feature that makes sure that even if you have multiple NotesDocument objects in scope in your code if they represent the same On-Disk document then there is only a single "backend" object used to reference the note. This feature prevents all kinds of data synch problems between multiple in-memory copies of a note and it's on-disk representation, very nice, but...

The NotesDocument Object contains a reference to the View/Collection from which it was derived, so if you have two collections open and both contain the same on-disk notes and you use GetNextDocument(doc) to navigate you stand a good chance (inevitable in the program example above) of tripping over the Gotcha, you have been warned.

LotusScript Uncovered Second Attempt


A minor re-coding of the Agent replacing the loop that read through the response documents with the following.

           If docTarget.Responses.Count > 0 Then
               For iIndex = 1 To docTarget.Responses.Count
                   Set docResponse = docTarget.Responses.GetNthDocument(iIndex)
                   iResps iResps + 1
               Next
           End If


The agent caould now run through without error and the CLIENT_CLOCK trace could be examined. The following is taken from the start of the trace.

(32-76 [32]) OPEN_DB(CN=Soho1/OU=Server/O=DeltaPhi!!HMI\Dev\ResLib.nsf): 10 ms. [134+290=424]
(33-76 [33]) GET_MODIFIED_NOTES(REP80257098:0034214D): 0 ms. [26+62=88]
(34-76 [34]) OPEN_NOTE(REP80257098:0034214D-NT000008FA,07400002): 10 ms. [48+4230=4278]
(35-76 [35]) OPEN_NOTE(REP80257098:0034214D-NT000008FA,03401001): 0 ms. [48+892”0]
(36-76 [36]) OPEN_NOTE(REP80257098:0034214D-NT000008FA,03401001): 10 ms. [48+892”0]
(37-76 [37]) OPEN_NOTE(REP80257098:0034214D-NT000008FA,03401001): 10 ms. [48+892”0]
(38-76 [38]) OPEN_NOTE(REP80257098:0034214D-NT0000090A,07400002): 0 ms. [48+4124=4172]
(39-76 [39]) OPEN_NOTE(REP80257098:0034214D-NT00000902,07400002): 10 ms. [48+4096=4144]
(40-77 [40]) OPEN_NOTE(REP80257098:0034214D-NT00000902,03401001): 10 ms. [48+810=858]
(41-77 [41]) OPEN_NOTE(REP80257098:0034214D-NT00000906,07400002): 10 ms. [48+4006=4054]
(42-77 [42]) OPEN_NOTE(REP80257098:0034214D-NT00000906,03401001): 0 ms. [48+830=878]
(43-77 [43]) OPEN_NOTE(REP80257098:0034214D-NT0000090E,07400002): 10 ms. [48+4128=4176]
(44-77 [44]) OPEN_NOTE(REP80257098:0034214D-NT0000090E,03401001): 0 ms. [48+810=858]
(45-77 [45]) OPEN_NOTE(REP80257098:0034214D-NT00000912,07400002): 0 ms. [48+3936=3984]
(46-77 [46]) OPEN_NOTE(REP80257098:0034214D-NT00000912,03401001): 10 ms. [48+828=876]
(47-77 [47]) OPEN_NOTE(REP80257098:0034214D-NT00000912,03401001): 0 ms. [48+828=876]
(48-77 [48]) OPEN_NOTE(REP80257098:0034214D-NT00000912,03401001): 10 ms. [48+828=876]
(49-77 [49]) OPEN_NOTE(REP80257098:0034214D-NT0000098E,07400002): 20 ms. [48+16632=16680]
(50-77 [50]) OPEN_NOTE(REP80257098:0034214D-NT00000912,03401001): 0 ms. [48+828=876]
(51-77 [51]) OPEN_NOTE(REP80257098:0034214D-NT000009FE,07400002): 10 ms. [48+4094=4142]
(52-77 [52]) OPEN_NOTE(REP80257098:0034214D-NT00000916,07400002): 10 ms. [48+4658=4706]
(53-77 [53]) OPEN_NOTE(REP80257098:0034214D-NT00000916,03401001): 10 ms. [48+878’6]
(54-77 [54]) OPEN_NOTE(REP80257098:0034214D-NT0000091A,07400002): 10 ms. [48+4742=4790]
(55-77 [55]) OPEN_NOTE(REP80257098:0034214D-NT0000091A,03401001): 10 ms. [48+784=832]
(56-77 [56]) OPEN_NOTE(REP80257098:0034214D-NT0000091E,07400002): 10 ms. [48+3930=3978]
(57-77 [57]) OPEN_NOTE(REP80257098:0034214D-NT0000091E,03401001): 0 ms. [48+780=828]


The first document read (NT000008FA) is a main document that has a single response document (NT0000090A). The first thing that is apparent from the trace is that in LotusScript the access to the response document is done through only OPEN_NOTE RPC calls there is no view walking involved. The second thing that is apparent is that there seems to be rather a large number of re-reads of the main document before the response document can be accessed. The re-reads of the main document all use the same combination of open flags.

OPEN_FLAGS = 0x03401001

From the C-API nsfnote.h file the open flags are interpretted as :-

0x02000000  =  OPEN_RAW_MIME_PART  
0x01000000  =  OPEN_RAW_RFC822_TEXT
0x00400000  =  Undocumented flag
0x00001000  =  OPEN_RESPONSE_ID_TABLE
0x00000001  =  OPEN_SUMMARY

This was enough information for me to be able to recall how I had done the reading of response documents in the past.

The C-API Solution


Open the main document note specifying OPEN_RESPONSE_ID_TABLE in the open flags (this flag can be used with both NSFNoteOpen and NSFNoteOpenExt), the flag causes the response noteid table to be included in the document header information that is returned from the call, and is in fact documented in the C-API documentation but only in the Symbolic Values section for the OPEN_XXX (note) flags.

OPEN_RESPONSE_ID_TABLE          -  Generate an ID Table of Note IDs for the responses to this note. Use this option in order to access the Note IDs of the immediate responses to a given note in a later call to NSFNoteGetInfo() using _NOTE_RESPONSES as the note header member ID.

In fact the documentation specifies how the ID table can be accessed using NSFNoteGetInfo() and specifying the _NOTE_RESPONSES as the member ID. The C-API documentation on the note header member flags also contains useful information.

    _NOTE_RESPONSE_COUNT          -  Get the number of immediate response notes for this note (DWORD).
    _NOTE_RESPONSES          -  Get a handle to the ID Table of Note IDs of the immediate responses of this note (HANDLE). If a note has no responses, NULLHANDLE is returned. There is no ordering of the response Note IDs in the table. The OPEN_RESPONSE_ID_TABLE flag must be specified when the note is opened in order to obtain a valid ID Table handle. If the OPEN_RESPONSE_ID_TABLE flag is not specified when the note is opened, NULLHANDLE will be returned as the value of the ID Table handle. Do not explicityly deallocate the ID Table. It will be deallocated by NSFNoteClose().

Armed with this information it was then possible for me to re-write my C-API function in a much more compact and efficient manner than the previous implementation. The following code snippets show the essential steps.


stErrorCode = NSFNoteOpenExt(hdbReport, nidList[dwIndex], OPEN_EXPAND | OPEN_RESPONSE_ID_TABLE, &hnDEInfo);
.
.
NSFNoteGetInfo(hnDEInfo, _NOTE_RESPONSES, &hidtDEL);
.
bool bFirstFlag2 = true;
while (IDScan(hidtDEL, bFirstFlag2, &nidEResponse))
{
   bFirstFlag2 = false;
   //  Open the response document
   stErrorCode = NSFNoteOpenExt(hdbReport, nidEResponse, OPEN_EXPAND, &hnELink);
   .
   .
   //  Process the response document
}


LotusScript Gotcha #2


Another point that was noticed from the CLIENT_CLOCK trace from the LotusScript agent was high number of "re-reads" of the main document that were needed to be able to access the response documents.
It would appear that every reference to the response collection forces a re-read of the document to re acquire the collection i.e. the response collection is not buffered. This could, of course, have a significant and unexpected performance penalty on application code. One simple thing that can be done to mitigate this effect would be to read the collection count into a local variable and only reference it from that variable.

The Java Story


Repeating the LotusScript tests in Java revealed that 1) Gotcha #1, the "Document is not from this collection" gotcha also applies to Java and 2) the references to the Responses count and collection also cause additional reads of the main document. In summary it behaves just the same as the LotusScript tests. Below is the Java test code and the starting portion of the CLIENT_CLOCK trace.

import lotus.domino.*;

public class JavaAgent extends AgentBase {
   public void NotesMain() {
       try {
           Session sessCurrent = getSession();
           AgentContext actxCurrent = sessCurrent.getAgentContext();
           int iDocs = 0;
           int iResps = 0;
           Database dbTarget = sessCurrent.getDatabase("soho1/Server/DeltaPhi", "HMI/Dev/ResLib.nsf");
           if (!dbTarget.isOpen())
           {
               System.out.println("ERROR: Failed to open target database.");
               return;
           }
           DocumentCollection dcAll = dbTarget.getAllDocuments();
           if (dcAll == null)
           {
               System.out.println("ERROR: Could not get the All Docs collection");
               dbTarget.recycle();
               return;
           }
           if (dcAll.getCount() == 0)
           {
               System.out.println("ERROR: All docs collection is empty.");
               dcAll.recycle();
               dbTarget.recycle();
               return;
           }
           Document docTarget = dcAll.getFirstDocument();
           while (docTarget != null)
           {
               if (!docTarget.isResponse())
               {
                   iDocs++;
                   if (docTarget.getResponses().getCount() > 0)
                   {
                       int iIndex;;
                       for (iIndex = 1; iIndex <= docTarget.getResponses().getCount(); iIndex++)
                       {
                           iResps++;
                           Document docResponse = docTarget.getResponses().getNthDocument(iIndex);
                           docResponse.recycle();
                       }
                   }
               }
               Document docOldTarget = docTarget;
               docTarget = dcAll.getNextDocument(docTarget);
               docOldTarget.recycle();
           }
           System.out.println("INFO: Scan completed, " + iDocs + " documents read and " + iResps + " responses read.");
       } catch(Exception e) {
           e.printStackTrace();
       }
   }
}


(32-122 [32]) OPEN_DB(CN=soho1/OU=Server/O=DeltaPhi!!HMI\Dev\ResLib.nsf): (Connect to Soho1/Server/DeltaPhi: 140 ms) (Exch names: 0 ms)(Authenticate: 0 ms.)
(OPEN_SESSION: 10 ms)
0 ms. [134+290=424]
(33-122 [33]) GET_MODIFIED_NOTES(REP80257098:0034214D): 90 ms. [26+62=88]
(34-122 [34]) OPEN_NOTE(REP80257098:0034214D-NT000008FA,07400002): 0 ms. [48+4230=4278]
(35-122 [35]) OPEN_NOTE(REP80257098:0034214D-NT000008FA,03401001): 10 ms. [48+892”0]
(36-122 [36]) OPEN_NOTE(REP80257098:0034214D-NT000008FA,03401001): 0 ms. [48+892”0]
(37-122 [37]) OPEN_NOTE(REP80257098:0034214D-NT000008FA,03401001): 0 ms. [48+892”0]
(38-122 [38]) OPEN_NOTE(REP80257098:0034214D-NT0000090A,07400002): 0 ms. [48+4124=4172]
(39-122 [39]) OPEN_NOTE(REP80257098:0034214D-NT000008FA,03401001): 10 ms. [48+892”0]
(40-122 [40]) OPEN_NOTE(REP80257098:0034214D-NT00000902,07400002): 0 ms. [48+4096=4144]
(41-122 [41]) OPEN_NOTE(REP80257098:0034214D-NT00000902,03401001): 10 ms. [48+810=858]
(42-122 [42]) OPEN_NOTE(REP80257098:0034214D-NT00000906,07400002): 0 ms. [48+4006=4054]
(43-122 [43]) OPEN_NOTE(REP80257098:0034214D-NT00000906,03401001): 10 ms. [48+830=878]
(44-122 [44]) OPEN_NOTE(REP80257098:0034214D-NT0000090A,07400002): 0 ms. [48+4124=4172]
(45-122 [45]) OPEN_NOTE(REP80257098:0034214D-NT0000090E,07400002): 10 ms. [48+4128=4176]
(46-122 [46]) OPEN_NOTE(REP80257098:0034214D-NT0000090E,03401001): 0 ms. [48+810=858]
(47-122 [47]) OPEN_NOTE(REP80257098:0034214D-NT00000912,07400002): 0 ms. [48+3936=3984]
(48-122 [48]) OPEN_NOTE(REP80257098:0034214D-NT00000912,03401001): 10 ms. [48+828=876]
(49-122 [49]) OPEN_NOTE(REP80257098:0034214D-NT00000912,03401001): 0 ms. [48+828=876]
(50-122 [50]) OPEN_NOTE(REP80257098:0034214D-NT00000912,03401001): 10 ms. [48+828=876]
(51-122 [51]) OPEN_NOTE(REP80257098:0034214D-NT0000098E,07400002): 10 ms. [48+16632=16680]
(52-122 [52]) OPEN_NOTE(REP80257098:0034214D-NT00000912,03401001): 10 ms. [48+828=876]
(53-122 [53]) OPEN_NOTE(REP80257098:0034214D-NT00000912,03401001): 10 ms. [48+828=876]
(54-122 [54]) OPEN_NOTE(REP80257098:0034214D-NT000009FE,07400002): 0 ms. [48+4094=4142]
(55-122 [55]) OPEN_NOTE(REP80257098:0034214D-NT00000912,03401001): 10 ms. [48+828=876]
(56-122 [56]) OPEN_NOTE(REP80257098:0034214D-NT00000916,07400002): 0 ms. [48+4658=4706]
(57-122 [57]) OPEN_NOTE(REP80257098:0034214D-NT00000916,03401001): 10 ms. [48+878’6]
(58-122 [58]) OPEN_NOTE(REP80257098:0034214D-NT0000091A,07400002): 0 ms. [48+4742=4790]
(59-122 [59]) OPEN_NOTE(REP80257098:0034214D-NT0000091A,03401001): 0 ms. [48+784=832]
(60-122 [60]) OPEN_NOTE(REP80257098:0034214D-NT0000091E,07400002): 10 ms. [48+3930=3978]
(61-122 [61]) OPEN_NOTE(REP80257098:0034214D-NT0000091E,03401001): 0 ms. [48+780=828]


Comments