Hi Enrico, Referring the testing:
I already had a deeper look into the implemented tests. I hope you don’t mind if I write you, why I think that the way how it is implemented should be improved.
no problem with constructive criticism.
For historical reasons, I wrote short test functions when I needed them, without having a full testing framework in my mind 15 years ago. 1. Fast feedback on where is what broken (the traffic light system on unit tests)
But this requires an IDE with GUI, doesn't it? Or, do you mean a message on console: OK/GREEN or ERROR/RED?
Here is an impression how it will look like. This is the context of the produced log file. I added a test which will fail. I “converted” two existing test-classes in the proposed directory-structure. Start testing: Mar 07 15:22 JST ---------------------------------------------------------- 1/2 Testing: constant_test 1/2 Test: constant_test Command: "/Users/enrico/Projects/cybop2/test/constant_test" Directory: /Users/enrico/Projects/cybop2 "constant_test" start time: Mar 07 15:22 JST Output: ---------------------------------------------------------- <end of output> Test time = 0.10 sec ---------------------------------------------------------- Test Passed. "constant_test" end time: Mar 07 15:22 JST "constant_test" time elapsed: 00:00:00 ----------------------------------------------------------
2/2 Testing: variable_test 2/2 Test: variable_test Command: "/Users/enrico/Projects/cybop2/test/variable_test" Directory: /Users/enrico/Projects/cybop2 "variable_test" start time: Mar 07 15:22 JST Output: ---------------------------------------------------------- Assertion failed: (1 == 2 && "obviously it is not"), function test_integral_type_sizes, file /Users/enrico/Projects/cybop2/test/variable/variable_tester.c, line 52. <end of output> Test time = 2.60 sec ---------------------------------------------------------- Test Failed. "variable_test" end time: Mar 07 15:22 JST "variable_test" time elapsed: 00:00:02 ----------------------------------------------------------
End testing: Mar 07 15:22 JST I used a small hack for having some kind of information if the assert is not obviously enough. unfortunately the assert in c is quite restricted, but maybe it will serve our purpose. Instead of using another framework, we could probably just provide some easy to use assert methods. I have seen plenty of prints in the unit tests you wrote, which are printing the result on the screen, this will not lead to an easy overview on where is what broken. Instead it is necessary to know the tests and to read all the output in detail. This is time consuming and probably frustrating. Rather than using prints, tests should fail if an expected value does not equal with the actual one. This should be improved, either by using asserts (i really don’t recommend this, there is no good feedback possible) or another framework which supports this easily
I understand your comment and know TDD, of course.
I’m not talking about TDD so much. Just making the feedback of the tests more efficient. However, a new framework brings new dependencies and overhead.
I don't say: "no", but have to be convinced by the framework.
The source should not get polluted by test functions.
Ideally, they should be bundled in an own directory, e.g. src/test/
I understand your concern and resigned from using another framework. To be honest, the c unittest frameworks are just convenient due to the assert methods they offer. But to create suites and tests we can use CTest, which seems to make a lot of things possible already. I really like the CTest. Impressive framework. 2. testing unit’s rather than integration
Looks like that most of the tests are integration tests, they are using several parts of the program, which will easily lead to issues when changing the program. Rather than testing processes, unit’s should be tested.
Puhh, this is quite abstract. CYBOI is not OOP, just structured-procedural programming. Nearly every CYBOI function is encapsulated in its own source file/module.
I'm not sure how you want to create or use units,
but go ahead and give it a try and us an example.
Sorry, maybe i was a bit to abstract. With units i meant the smallest element in the program. Methods like compare_integer_equal. e.g. Please find two examples in the test-directory i created. You can revert the commit anytime if you like. Of course i used the one’s which have been easiest to convert into the new structure. Just to get an idea. You just “complained" in a former email that the unit tests became obsolete because the structure changed a lot. I just wanted to state that it is also due to the reason that most of the tests are integration tests and use a lot of functions from all over. Which of course needs a lot of time to maintain, e.g. modifier_test.
3. Connected with the source
I think it would be better to exclude the tests from the source. This way it will not be contained in the binary. A second advantage of the separation would be that it is not necessary to adjust the tester.c all the times tests are added or removed. Using the current approach will create a lot of code which needs to maintained.
O.k., I agree. If you find a good solution to do this, go ahead. 4. automatic execution
CTest would allow to execute all defined tests and also create some dashboard for code-coverage results and so on. As far as i understand the structure how it is at the moment, can’t be easily integrated into CTest.
O.k. Proposal:
1. Extracting the tester-directory out of the src directory, to e.g. tests
I agree. Is it possible to call it in singular, i.e. "test/". Please create this new directory in the CYBOP project's root,
Done, please have a look! i.e. parallel to "src/" and "examples/".
("examples/" will also be named in singular one day)
2. Putting the classes into a src similar folder structure under tests.
Not useful. CYBOI consists of a number of directories. The following are not testable, as they contain just values: - constant - variable The logger is global and probably does not need special testing: - logger The tests themselves will be moved away from "src/" in the future: - tester The applicator just delegates function calls with one function: - applicator Not really useful to test this, as it could be done in CYBOL/XML as well. The controller is not really testable: - controller It contains the signal/event processing loop, IRQ mechanisms, threads. The ONLY directory I consider "testable" is: - executor Here are the basic functions, arithmetic, comparison, deserialiser etc. Addition: In the "test/" directory, I have written some functions to understand the functioning of the C programming language, e.g. pointers. These files and functions are not related to CYBOI or unit tests. Probably, they would have to be factored out into documentation then.
Ah, I see. Yeah, or just keep it as “unittest”. 3. rewrite the tests to use some proper mechanism to fail when expected != actual
If you'd like to do this work, then please go ahead! 4. integrate the tests to CTest
O.k. If this is possible, then do it. I am eager to know what you think about the proposal.
A student from IT 2011 had the same idea and copied the whole "src/" structure 1:1. In the end, it was not really helpful or necessary and cost me a lot of time to clean up and roll back a lot of stuff, in order to keep overview.
That’s not what i meant by having a similar structure. It should be more abstract and not that detailed as the src. It is just an approach to find tests belonging to a specific class/file earlier. Having hundred of test-“classes" lying around in one folder doesn’t make it easy to directly get to the unittest someone want’s to work on.
My suggestion for you:
1 Focus on CMake first, until it works.
2 Afterwards, try to implement (only) one or two unit tests.
I did it already :)
If all turns out to be usable and work fine, then you may stepwise
extend unit tests and thereby understand the CYBOI architecture.
Then we'll see further.
What do you think?
What do you think? :) Enrico
|