Bugzilla – Bug 677789
kiwi: integrate automation testsuite written by Bernhard Wiedemann
Last modified: 2011-05-09 09:44:36 UTC
I uploaded a first alpha of my os-autoinst-based kiwi testsuite into http://gitorious.org/~bmwiedemann/kiwi/bmwiedemanns-kiwi this includes http://gitorious.org/~bmwiedemann/kiwi/bmwiedemanns-kiwi/blobs/master/tests/os-a utoinst.kiwi/README The tests need kvm-0.13+ installed (only for the zypper test part, 0.12.x is good otherwise). One important missing thing is the checklog tool, where I need to do some improvement on the os-autoinst side to allow it to delegate the checks to distri-specific code. > Sounds great. Is the branched version of kiwi with your test > framework ready to become included or do we need more time. I think > we need more time because some doc hint seems to be required as > well. you can merge it into kiwi, as it does not break anything. Of course, it still needs more work to be done, to be more useful. > Is the os-autoinst framework part of the testing framework or > is this something different ? http://www.os-autoinst.org/ - it's git repo is cloned by the setup-script, and used by the run-script so it is sort of an external dependency for the kiwi tests. --- 08.03.2011 Today I have pushed updates to http://gitorious.org/~bmwiedemann/kiwi/bmwiedemanns-kiwi Together with updated os-autoinst that allows to do a fully automated testrun on a kiwi-generated 11.3 jeos iso. This creates a computer-readable test result summary.
Could you tell me the status on the packages front for the test suite ? any schedule available ? Thanks
closed no info
I have packaged os-autoinst as RPM at https://build.opensuse.org/package/show?package=os-autoinst&project=home%3Abmwiedemann%3AQA and just submitted it to https://build.opensuse.org/package/show?package=os-autoinst&project=devel%3AopenSUSE%3AFactory It currently does not include the kiwi testing part but that can be symlinked as the kiwi-test's setup.sh does.
thanks. I saw the packages exists for 11.4 only at the moment. Do you plan to support at least SLE11 and 11.3 too ? I'm asking because we are in the loop to create SLE11-SP2 and for local testing it would be great not to be forced to update the workstation to 11.4 ;) > It currently does not include the kiwi testing part > but that can be symlinked as the kiwi-test's setup.sh does. I plan to hold , write, update the tests itself in the kiwi repo and provide the test description together with the kiwi image description. It would be great if your test-suite can provide a tool which allows me to call something like: os-autotest /path/to/os-image-file /path/to/test-case and then the test runs. Does something like this exist ? if not could you add that ? and last but not least; Do you plan to write some documentation for your test-suite, maybe available in the wiki or another place I can point people to if they want to know more about writing the tests ? Hope I did not bother you too much Thanks
I enabled packaging of os-autoinst for SLE11-SP1. Since it is a noarch package, it should be usable on most rpm-based distributions. I started to collect some information for test-developers at http://www.os-autoinst.org/testmodules.html Since os-autoinst is not limited to openSUSE, the openSUSE wiki would IMHO not be the right place for such docs. with the rpm you can do isotovideo /path/to/os-image.iso but the test cases can not run stand-alone, because some depend on earlier steps (e.g. install). What you can do is use environment vars and the is_applicable method to enable or skip certain tests. See consoletest.d/600_mozmill_setup.pm as example. you're welcome
Hmm, when I look at the package I found for example: distri/sles-10/ check.pm inst.d main.pm test.d for kiwi I don't need anything in inst.d/ and on a short look it should be possible to write main.pm in a way that it does not install anything. So if a kiwi image description provides all that data I should be able to run a test with your test-suite, right ? if so, how would I start the test with start.pl ? and is there a specific reason why the image format must be an iso ? Thanks
The iso file is passed to kvm with -cdrom $iso because most of the non-kiwi testruns are centered around isos containing the software to test. It is possible to generalize that to also include HDD images that can be booted in qemu/kvm. it is probably possible to start&kill a new VM for each test. But it needs extra time for the repeated BIOS/bootloader/kernel/init part. And it might need some code changes. but you can also run DISTRI=manual ISO=/path/to/os-image.iso ./start.pl to bring up the VM and provide VNC for manual interaction and a TCP port (default:15223) for script interaction with it. This part will exit after 100 minutes or upon VM inactivity. Then you still need another script to run single tests. You could set the env var QEMUPORT=15223 and use the normal code paths for sending commands from stand-alone tests. And you need a way to decide if the outcome was OK/fail/unknown possibly also using the basetest::check method. Some kiwi image types do need an install step, but that could be included in the start procedure.
> wenn ich das Ergebnis des testlaufs anschaue bekomme ich > in meinem Fall folgendes: > > ==== > bootloader: not-autochecked > wait_booted: not-autochecked > change_tty2: OK-11.4 > login: unknown > login1: unknown > shutdown: not-autochecked > isosize: OK > overall: fail > > * was bedeutet "not-autochecked" ? > * wie kann ich herausfinden warum zB bei login der status 'unknown' ist > den login hat er gemacht wie im vncviewer gesehen habe > * warum ist overall 'fail' ? > 1. not-autochecked is normal. Das heisst nur, dass der Test nix weiter prueft. 2. unknown ist das Ergebnis, was rauskommt, wenn keiner der bekannt guten(OK) oder bekannt schlechten(fail) MD5 Werte oder screenshots(testimgs/) gesehen wurde. Wenn du ein screenshot hast, der ein definitiv richtiges Ergebnis zeigt, dann fuegst du entweder die md5sum des ppm files zu tests/os-autoinst.kiwi/iso/test.d/300_login.pm oder aber eine MD5sum von einem vordefinierten Ausschnitt (die Werte, die tools/inststagedetect2.pl $ppm liefert). 3. Das overall wird in distri/kiwi.oem/check.pm bestimmt und ist nur OK, wenn keine schlechten Ergebnisse vorkommen, wozu die "unknown"s aber zaehlen. Kann man natuerlich auch anders definieren (z.B. bei Factory tests reicht es, wenn zypper, yast und (xterm oder firefox) gingen). fuer das ogg/theora encoden nehm ich auf SLES11 http://openqa.opensuse.org/opensuse/tools/umlffmpeg http://openqa.opensuse.org/opensuse/tools/dummypiper http://openqa.opensuse.org/opensuse/tools/linux-2.6.36 nach /usr/local/bin/ , chmod a+x und http://openqa.opensuse.org/opensuse/img/lenny32.img.gz auspacken die Pfade in umlffmpeg anpassen und geht
I have added support for the os-autoinst test suite now. I added one test example to the 11.4 JeOS image description... more will follow. To run the test the following kiwi command is required kiwi --test-image /tmp/mytest/LimeJeOS-openSUSE-11.4.i686-1.11.4.iso \ --test-case /usr/share/kiwi/image/suse-11.4-JeOS/test/ \ --type iso May-09 11:40:19 <1> : Starting image test run... done May-09 11:40:19 <1> : Calling isotovideo, this can take some time... May-09 11:40:19 <1> : watch the screen session by: 'screen -r' done May-09 11:42:23 <1> : Find test results in /tmp/kiwi-testrun-EyyqgA/currentautoinst-log.txt support for other than iso image types would be great thanks