Bug 677789

Could you tell me the status on the packages front for the test suite ?
any schedule available ?

Thanks

closed no info

I have packaged os-autoinst as RPM at
https://build.opensuse.org/package/show?package=os-autoinst&project=home%3Abmwiedemann%3AQA

and just submitted it to
https://build.opensuse.org/package/show?package=os-autoinst&project=devel%3AopenSUSE%3AFactory

It currently does not include the kiwi testing part
but that can be symlinked as the kiwi-test's setup.sh does.

thanks. I saw the packages exists for 11.4 only at the moment. Do you plan
to support at least SLE11 and 11.3 too ? I'm asking because we are in the
loop to create SLE11-SP2 and for local testing it would be great not to
be forced to update the workstation to 11.4 ;)

> It currently does not include the kiwi testing part
> but that can be symlinked as the kiwi-test's setup.sh does.

I plan to hold , write, update the tests itself in the kiwi repo
and provide the test description together with the kiwi image
description. It would be great if your test-suite can provide a tool
which allows me to call something like:

   os-autotest /path/to/os-image-file /path/to/test-case

and then the test runs. Does something like this exist ? if not
could you add that ?

and last but not least; Do you plan to write some documentation for
your test-suite, maybe available in the wiki or another place I can
point people to if they want to know more about writing the tests ?

Hope I did not bother you too much

Thanks

I enabled packaging of os-autoinst for SLE11-SP1. Since it is a noarch package, it should be usable on most rpm-based distributions.


I started to collect some information for test-developers at
http://www.os-autoinst.org/testmodules.html


Since os-autoinst is not limited to openSUSE, the openSUSE wiki would IMHO not be the right place for such docs.

with the rpm you can do
   isotovideo /path/to/os-image.iso

but the test cases can not run stand-alone, because some depend on earlier steps (e.g. install).
What you can do is use environment vars and the is_applicable method to enable or skip certain tests. See consoletest.d/600_mozmill_setup.pm as example.

you're welcome

Hmm, when I look at the package I found for example:

  distri/sles-10/

         check.pm
         inst.d
         main.pm
         test.d

for kiwi I don't need anything in inst.d/ and on a short look
it should be possible to write main.pm in a way that it does not
install anything.

So if a kiwi image description provides all that data I should
be able to run a test with your test-suite, right ?

if so, how would I start the test with start.pl ?

and is there a specific reason why the image format must be an iso ?

Thanks

The iso file is passed to kvm with -cdrom $iso because most of the non-kiwi testruns are centered around isos containing the software to test. It is possible to generalize that to also include HDD images that can be booted in qemu/kvm.


it is probably possible to start&kill a new VM for each test. But it needs extra time for the repeated BIOS/bootloader/kernel/init part. And it might need some code changes.


but you can also run 
DISTRI=manual ISO=/path/to/os-image.iso ./start.pl
to bring up the VM and provide VNC for manual interaction and a TCP port (default:15223) for script interaction with it.
This part will exit after 100 minutes or upon VM inactivity.

Then you still need another script to run single tests. You could set the env var QEMUPORT=15223 and use the normal code paths for sending commands from stand-alone tests.
And you need a way to decide if the outcome was OK/fail/unknown possibly also using the basetest::check method.

Some kiwi image types do need an install step, but that could be included in the start procedure.

> wenn ich das Ergebnis des testlaufs anschaue bekomme ich
> in meinem Fall folgendes:
>
> ====
> bootloader: not-autochecked
> wait_booted: not-autochecked
> change_tty2: OK-11.4
> login: unknown
> login1: unknown
> shutdown: not-autochecked
> isosize: OK
> overall: fail
>
> * was bedeutet "not-autochecked" ?
> * wie kann ich herausfinden warum zB bei login der status 'unknown' ist
>   den login hat er gemacht wie im vncviewer gesehen habe
> * warum ist overall 'fail' ?
>

1. not-autochecked is normal. Das heisst nur, dass der Test nix weiter
prueft.

2. unknown ist das Ergebnis, was rauskommt, wenn keiner der bekannt
guten(OK) oder bekannt schlechten(fail) MD5 Werte oder
screenshots(testimgs/) gesehen wurde.
Wenn du ein screenshot hast, der ein definitiv richtiges Ergebnis zeigt,
dann fuegst du entweder die md5sum des ppm files zu
tests/os-autoinst.kiwi/iso/test.d/300_login.pm oder aber eine MD5sum von
einem vordefinierten Ausschnitt
(die Werte, die tools/inststagedetect2.pl $ppm liefert).

3. Das overall wird in distri/kiwi.oem/check.pm bestimmt und ist nur OK,
wenn keine schlechten Ergebnisse vorkommen, wozu die "unknown"s aber
zaehlen. Kann man natuerlich auch anders definieren (z.B. bei Factory
tests reicht es, wenn zypper, yast und (xterm oder firefox) gingen).


fuer das ogg/theora encoden nehm ich auf SLES11
http://openqa.opensuse.org/opensuse/tools/umlffmpeg
http://openqa.opensuse.org/opensuse/tools/dummypiper
http://openqa.opensuse.org/opensuse/tools/linux-2.6.36
nach /usr/local/bin/ , chmod a+x
und http://openqa.opensuse.org/opensuse/img/lenny32.img.gz auspacken
die Pfade in umlffmpeg anpassen und geht

I have added support for the os-autoinst test suite now. I added one test
example to the 11.4 JeOS image description... more will follow. To run the
test the following kiwi command is required

   kiwi --test-image /tmp/mytest/LimeJeOS-openSUSE-11.4.i686-1.11.4.iso \
        --test-case /usr/share/kiwi/image/suse-11.4-JeOS/test/ \
        --type iso


May-09 11:40:19 <1> : Starting image test run...                     done
May-09 11:40:19 <1> : Calling isotovideo, this can take some time...
May-09 11:40:19 <1> : watch the screen session by: 'screen -r'       done
May-09 11:42:23 <1> : Find test results in /tmp/kiwi-testrun-EyyqgA/currentautoinst-log.txt                                                          

support for other than iso image types would be great

thanks