sporadic false resource tracker alarms

[Ichthyostega]

NOBUG_RESOURCE_LEAVE happens after releasing the lock (sectionlock.h)
It is likely that another competing thread immediately aquires the lock when it gets available. In the observed cases, on thread termination, the other thread after unblocking proceeded with the dtor and pulled away the condition var, thus causing the alarm.

[Christian Thäter]

Yes, I had a glimpse of this problems, just didn't realized how often they can happen. Now while you point that out, this needs some fix, possibly changing the NoBug API creating a lock around this actions:

 NOBUG_RESOURCE_WAITING_ENTER(flag, resource, user, state, handle,
  ...code...
 )

this goes from waiting to state while executing '...code...' inbetween and keeping the resourcetracker lock held

and similar a

NOBUG_RESOURCE_LEAVE(flag, handle, ...code...)

which first locks the resource tracker, then executes '...code...' and then calls the leave and unlock the resourcetracker

[Christian Thäter]

fixed in nobug/devel but needs changes on the Lumiera code, i am working on this.

[Christian Thäter]

Fixed in:

​http://git.lumiera.org/gitweb?p=lumiera/ct;a=commit;h=5c7abbded99585d7334f075f57c1b43437b3909a

This breaks 40components.test "Type-based contexts" TypedCounter_test, which hangs from the testsuite here, this needs to be investigated.

[Ichthyostega]

Hi cehteh,

read your changes (including NoBug) and they look OK. I can't see why the test would hang.
Thus reopening and assigning this ticket to myself, as a reminder to investigate it
further with the debugger (after updating the NoBug package, which I'll do in some days)

For later referral, I'll reiterate the sequence of events:

• JoinHandle locks the mutex of the reccondition and handes it into the tread starter
• in the spawned child thread, after the end of the thread function, the thread starter

    if (!thread_end_notification)
      return NULL; // no signalling of thread termination desired
  
    LUMIERA_RECCONDITION_SECTION(cond_sync, thread_end_notification)
      LUMIERA_RECCONDITION_BROADCAST;
  

  ...which causes it to get blocked, as the thread_end_notification condition is already locked
• in the destructor of the managing class, which inherits from JoinHandle (see typed-counter-test.cpp):

        ~SingleCheck () { this->join(); }
  

  which is implemented by the "manager" thread going into wait state
• this has the sideeffect of releasing the lock, thus the child thread can now enter the LUMIERA_RECCONDITION_SECTION. It does the broadcast, but the manager thread still has to re-gain the lock
• now, the moment the LUMIERA_RECCONDITION_SECTION drops the lock, it could be prempted. Note, this is still in the middle of the LUMIERA_RECCONDITION_SECTION
• assumed this happens, now the manager thread can re-gain the lock and recieve the wakeup. This causes the destructor to continue and finish, thereby releasing the lock and then discarding the object including the condition variable.
• note this means lumiera_reccondition_destroy can be executed while the child thread is still preempted in the middle of the LUMIERA_RECCONDITION_SECTION. But as the resource tracking is now protected by a lock, this should cause no problem (?)
• similarily, as far as I can see, your modified code doesn't touch the lock or the condition anymore after releasing the lock; thus I'd expect it just to release and cleanup the nobug resource handle and then finish executing the child thread.

[Christian Thäter]

Replying to ichthyo:

Hi cehteh,

read your changes (including NoBug) and they look OK. I can't see why the test would hang.
Thus reopening and assigning this ticket to myself, as a reminder to investigate it
further with the debugger (after updating the NoBug package, which I'll do in some days)

For later referral, I'll reiterate the sequence of events:

• JoinHandle locks the mutex of the reccondition and handes it into the tread starter
• in the spawned child thread, after the end of the thread function, the thread starter

    if (!thread_end_notification)
      return NULL; // no signalling of thread termination desired
  
    LUMIERA_RECCONDITION_SECTION(cond_sync, thread_end_notification)
      LUMIERA_RECCONDITION_BROADCAST;
  

  ...which causes it to get blocked, as the thread_end_notification condition is already locked
• in the destructor of the managing class, which inherits from JoinHandle (see typed-counter-test.cpp):

        ~SingleCheck () { this->join(); }
  

  which is implemented by the "manager" thread going into wait state
• this has the sideeffect of releasing the lock, thus the child thread can now enter the LUMIERA_RECCONDITION_SECTION. It does the broadcast, but the manager thread still has to re-gain the lock
• now, the moment the LUMIERA_RECCONDITION_SECTION drops the lock, it could be prempted. Note, this is still in the middle of the LUMIERA_RECCONDITION_SECTION
• assumed this happens, now the manager thread can re-gain the lock and recieve the wakeup. This causes the destructor to continue and finish, thereby releasing the lock and then discarding the object including the condition variable.
• note this means lumiera_reccondition_destroy can be executed while the child thread is still preempted in the middle of the LUMIERA_RECCONDITION_SECTION. But as the resource tracking is now protected by a lock, this should cause no problem (?)
• similarily, as far as I can see, your modified code doesn't touch the lock or the condition anymore after releasing the lock; thus I'd expect it just to release and cleanup the nobug resource handle and then finish executing the child thread.

I am going over some more nobug things right now, next I want to refactor the lumiera locking (mutex/condition/rwlock). While Plouj now works on the threadpool it becomes imminent that the sync.hpp uses the resourcetracker too. I am thinking about factoring a lumiera_mutex_lock and lumiera_mutex_unlock function which includes the resourcetracking (and the same for condition vars and rwlocks). Then sync.hpp can use those and then may_block() enter() left() methods there can bite the dust.

[Christian Thäter]

btw, centralizing the locking at one place again will also make the lock profiling easier (contention timing...)