TIP 383: Injecting Code into Suspended Coroutines

Login
Home Timeline Branches Tags Help 
Author:         Alexandre Ferrieux <[email protected]>
Author:         Miguel Sofer <[email protected]>
Author:         Donal K. Fellows <[email protected]>
State:          Final
Type:           Project
Vote:           Done
Created:        03-Dec-2010
Post-History:   
Keywords:       debugging,coroutine,yielded
Tcl-Version:    8.7
Tcl-Branch:     tip-383
Votes-For:      DKF, KBK, JN
Votes-Against:  none
Votes-Present:  FV, SL

Abstract

This proposes two new commands. One, coroinject, that allows a programmer to inject arbitrary code into a suspended coroutine, for execution on next resumption, and the other, coroprobe, that allows a programmer to inject arbitrary code into a suspended coroutine for immediate execution (making it much easier to use for introspection).

Rationale

When debugging complex coroutines - with many yield points and possibly rich state in local variables - sometimes one would like to inspect their state “from the outside”, i.e., at a point where they are suspended.

A typical situation is that of a big, single-threaded, event+coro system, where the coro happily enables/disables fileevents along its life, and the fileevents are one way to resume the coro. At a given point (bug), things get stalled, with the fileevents disabled. The obvious questions are:

  1. “where” is the coro (at which yield site)? and

  2. what are the values of its local variables?

Both these questions can be answered with the new coroinject primitive. The idea is to force a resumption of the coro along with an “immediate execution of extra code” directive, where the extra code says “dump the call stack with info level and info frame”, or “dump the locals”, etc.

Another use would be to inject "return -code return", as an alternative to renaming to {} for terminating the coro in a way that respects its catch/finally termination handlers. Alternatively, returning with an error code will have the effect of gathering call stack information in the -errorstack options dictionary entry.

At the other end of the spectrum, the injected code can be completely transparent: either with a forced resumption and injected code ending with yield, or merely waiting for normal resumption when the app sees fit, and injected code falling back to normal coro code.

Note that the feature is similar to a proc-entry trace, but coroutine resumption is not currently a trace target. Also, it is an intrinsically "one-shot" mechanism, which makes it a better fit for its debugging purposes.

To make the use of coroinject simpler for the common introspection use case, an extra command, coroprobe is included. That runs the code immediately and produces the results immediately, including making errors in the running of the code not appear inside the coroutine; faults in debugging code should not cause failures in the code being debugged.

Definition of coroinject

The new command:

coroinject coroname cmd ?arg1 ...?

prepends to the code to be executed on resumption of the currently suspended coroutine, coroname, the following code:

cmd arg1... yieldtype resumearg

where yieldtype is either yield or yieldto, depending on how the coroutine yielded (allowing generic injection handler commands to be used), and_resumearg_ is:

  1. the single argument passed to the resumption command coroname when yield is used by coroname. If there is no resumption argument, resumearg will be the empty string.

  2. the list of arguments passed to the resumption command coroname when yieldto is used by coroname.

In turn, the result from the execution of cmd will be seen by the coroutine's code as the result of yield/yieldto.

Note that:

  1. Resumption itself must be done separately, by calling coroname later,

  2. If coroinject is called several times on the same coroname before resuming it, the commands pile up in LIFO order.

  3. In combination, the appending of resumearg and the use of the result of cmd to provide the result of yield, will allow the following style of fully transparent injection:

        proc probe {x y type resumearg} {
        do things $x $y
        return $resumearg
    }
    coroinject C probe foo bar

    However, probing is more simply done with coroprobe.

Definition of coroprobe

coroprobe coroname cmd ?arg1 ...?

runs the command cmd (with the given arguments) immediately in the currently suspended coroutine, coroname, and gives the results of that execution as the results of the coroprobe command. Errors in cmd result in errors from coroprobe, and not errors in the coroutine, coroname; the return command cannot be used to make the coroutine terminate early either. After coroprobe has run, the state of the coroutine will be the same as before (assuming cmd does not delete coroname or alter any of its local variables); for example, if it was waiting for multiple arguments before (because it had used yieldto, it will continue to do so afterwards, and it will resume at the same point in the code as it would have if coroprobe had not be run.

Unlike with coroinject, no extra arguments are appended to cmd when it is executed beyond those suppled as part of the coroprobe call.

This combination of features means that it is possible to use this command usefully with, say, info frame and info level to examine the state of the coroutine.

    set f [coroprobe C info frame]
    puts "coroutine C @$f: [coroprobe C info frame $f]"

However, in general it is easier to put a more complex probe into a procedure or lambda term:

    set vars [coroprobe C apply {{} {
        lmap var [uplevel 1 info locals] {
            upvar 1 $var val
            string cat $var -> $val
        }
    }}]
    puts "The vars of C are: $vars"

Naming

Of course, the proposed coroinject is a placeholder for a suitable name (and similarly coroprobe). Alternatives that also make sense are: ::tcl::coroinject and interp coroinject. Constructive bikeshedding welcome.

Reference Implementation

See the tip-383 branch.

Notes

The current ::tcl::unsupported::inject implements most of the functionality described here, minus the yieldtype and resumearg passing. The yieldtype value is determinable with ::tcl::unsupported::corotype.

coroprobe can be mostly implemented using coroinject, but not easily. In particular, the error handling is non-trivial and info frame will be able to tell the difference consistently.

The coroprobe command is less general than coroinject, but is quite a bit easier to use by virtue of being immediate and able to pass errors out. However, there are use cases out there that would not be supported by coroprobe, such as the operations on the ‘every’ wiki page that use ::tcl::unsupported::inject to do nice early termination of the worker coroutines. Thus, having two commands is jusifiable. The unsupported version of this has problems with processing the resumption value, which isn't a problem when the resumption is done straight away (making its resumption value moot), but that in turn can have problems with knowing how to re-yield so as to not disrupt the coroutine, which is why the yieldtype value is added (to allow the injected code to yield correctly).

Copyright

This document has been placed in the public domain.