Hi everyone,
I am new to this mailing list, and writing this because I have encountered some issues with
ice-9 expect:
The procedural expect macro has a serious problem:
This syntax expects a procedure as the test _expression_, with 2 arguments (actual content read, and boolean EOF flag). The primary problem is that when the test _expression_ is not a identifier bound to such a procedure, but instead an _expression_ which creates such a procedure (aka "factory"), then the _expression_ will be evaluated anew for each character read from expect-port. This is a serious problem: when constructing that procedure has significant computational costs, or side effects (eg. opening a port to a new log file), as it can be seen in my code
here. (This is a branch of some of my KVM-based E2E testing code, which uses the original expect macro implementation, just to demonstrate the problem).
I had written a workaround for this problem, by binding the evaluation of the test _expression_ outside the expansion of the expect macro (as it can be seen
here). The problem with this was, that it doesn't support the full capabilities of the original expect macro, and debilitates it by only allowing for a single conditional branch. Also, I haven't verified, but I suspicious it would possibly even break the behaviour for the
=> syntax of the
expect-strings macro.
To implement a more complete version of this (and also as an exercise to understand hygienic macros), I've attempted to rewrite my workaround using
syntax-case. I've ran into some issues, and my implementation didn't expand as I expected. After some discussion on IRC, we've
concluded that this was because the current
(ice-9 expect) implementation is written using unhygienic
defmacro syntax, which doesn't work well together with the hygienic
syntax-case. It was suggested I should rather rewrite expect completely to be hygienic instead.
I've eventually completed the rewrite using syntax-case
here, and would be happy to contribute it back to the ice-9 module!
I would point out the following about this implementation:
- It works!!! I had some quite extensive E2E testing code written before here, which I think really stretches the possibilities with (ice-9 expect). Switching to the new implementation was simple, and works flawlessly!
- It is backwards compatible! I've gone extra lengths to make sure the code would be a backwards compatible, a drop-in replacement.
- Introduced new feature for the procedural expect macro, such that the second argument of the procedure gets passed the currently read char, or #f if it was an EOF character. There is multiple reasons this is superior to passing just a boolean eof? flag argument:
- the same logic can be implemented as with the eof? boolean, just by negating the condition
- passing
the currently read char avoids having to do the inefficient operation to retrieve the currently read char by cutting off the last character from the end of the content. Currently it is necessary to do this, if the logic wants something additional to be done with the current char (eg. as it
can be seen in my code here)
- one
such case of additional use is making the expect-char-proc property obsolete, as the predicate procedure is now able to execute such logic too, besides
other more advanced logging scenarios (like it can be seen in my code here:
here besides logging everything to STDOUT, and also to a main log-file, I
also log for each expect macro call the contents it tries to compare
against into a separately named minor log-file (useful for debugging), and also to ensure that
we actually do not write to STDOUT when using expect on multiple parallel
processes)
- To ensure that everything stays backwards compatible, this new feature is only enabled if the new expect-pass-char? binding is set to #t in the lexical context (like here). Otherwise, by default this feature is disabled, and boolean eof? flag argument is passed just like before.
- There is a better support for using default value/expressions for the various parameters the macros depend on in their lexical contexts. The old macro had to export default values defined in its module, so as to ensure the defaults are available. This either clutters the module's global context where (ice-9 expect) is used, or the defaults are not available when using something like ((ice-9 expect) #:select (expect expect-string)). The new defaults are calculated based on the lexical context where the macros are used, and expands into the correct default values when they are not defined. For example when expect-port is not defined in the lexical context, then the default value used is going to be the result of the (current-input-port) _expression_. I would like someone to review whether I've implemented this correctly, as it seems the current-input-port identifier default for expect-port gets captured in the expansion. Not sure why that happens.
- The expect-strings macro supports the => syntax, which passes the output(s) of the test predicate to a procedure, instead of just evaluating a body. This is fully supported, but additionally, the => syntax now also works with the procedural expect macro too! I haven't verified if the original implementation did this too, but at least it wasn't described in the documentation!
- As an added bonus, I have added a simplistic implementation for an interact macro. It uses threads to read from STDIO, and expects expect-port to be an input-output-port, so that it can output the lines read from STDIO into the same port. Not too advanced, as it doesn't support a full terminal interface, with autocomplete, etc... but works well enough for my use-case to interact with a KVM process via serial port.
- mnieper said that I am not using syntax-case idiomatically, and my style is more in the form of a syntax-rules-based state-machine.
I am quite happy with how it is currently, but would be open to be convinced if there is a superior way of expressing all this. - I currently have various tests sprinkled around my implementation, to manually demonstrate expansions, and executions.
Also, I have added some automated unit tests for the bind-locally helper procedure, which I used for checking whether parameters are defined in the lexical scope. - I am not happy with how I am managing my project structure, and how my modules can reference each other (having to call add-to-load-path everywhere).
Also, I think the way I am using srfi-64 tests is a bit bothersome, as they always get executed when my modules are loaded, and I have to globally disable logging to files.
I would like some recommendations how to do this better!
If the community likes this implementation, I would be happy to apply/rebase my changes onto the official repo, and then we can go from there!
Best regards,
Daniel Dinnyes