Improver (English version) — различия между версиями

Версия 20:18, 27 июня 2011

Most of the testing tasks for Mandriva/ROSA operating systems are to be done while running Improver, exept those which need data recollecting before running the system itself i.e. that concern installation and booting the system, as well as preparing bootable media and running system within virtual machines. For those kind of tests there are special protocols to be filled after performing the tests and recollecting the data. Improver is a kind of software dedicated to collection automatically the necessary data of the PC used for testing the system and running the tests as replies to questions or statements which the client software presents to a registered user according to testing plans downloaded from Improver's server. The whole Mandriva/Rosa's Improver includes

• client software
• server applications
• web-application

Client software is GUI utility called Improver_client. It creates user account, downloads testing plans from the server, and collects the hardware and system data from the user's PC. User selects desired testing plan form available list, fills report then the program sends it to the Improver's server.

Usage of Improver

User may run it from main menu --> "Applications" --> "Tests" or by the command `improver' in console. The root privileges are required because the program launches such system utilities as lshw and dmidecode to collect information about hardware.

Druing starting you can see splash screen, this time program is calculating hw_id and connecting with server. If there is now your account on server you will be asked to register. If you already have account you'll se your account information.

Next, window with common list of test plans is appears. Test plans are divided by groups. Each test plan has icon representing status of test plan: new, downloaded, started, finished, sent. At the bottom of screen are control buttons. Update - updating information about test plans. Map - representing questions map of test plan. Import - import test plan from XML file. Send reports - send saved reports. By clicking all stored on the workstation reports are sent to server. Start - begin to perform the selected test plan (also you can do it by double-clicking on row in test plans table).

After selecting test plan for performing opened window with description of test plan. Next, window with test plans questions. Under questions left side - answer options; right side - comments for question.

In different questions you can select one or more answers. There are customized keyboard shortcuts for quick access: Alt+1 .. Alt+9 and Alt+q for "other" answer.

Next below are buttons: screen capture, screen capture with delay, attach a file, map of test plan.

By clicking on the "Map" opens a window with a list of test plans questions. In the window indicate the status of the question - answered or not answered. From the window with the map of plan you can move to any question of test plan.

At the bottom of window are buttons: Test plans list - back to common list of test plans with savind current progress. Back - go to the previous question. Next - go to the next question. Finish - go to sending report.

After completing test plan user can add an additional comment to test plan and add custom file to report.

Next, by clicking button "Send report" report isgenerating and will be sent to server. Then you can proceed to perform other test plans.

The following hardware is said by developers of the Mandriva/Rosa systems as most required for testing the Release Candidates 2011:

Any laptops equipped with Nvidia Optimus graphic cards supporting soft as well as hard switchig video

Any laptops supporting AMD Avivo technology

Laptops equipped with Intel GMA500 grphic card

MacBooks

Any atypical netbooks

RADEON graphic cards: r800, r700, r600, r520 - series; NVidya: 200-500 series, old ones from 3-rd to 8-th generations( geforce 3/4 + 6***/7***/8***) and their mobile counterparts

Any graphic cards with HDMI out

SIS graphic cards

Multy-display systems

Popular audio adapters like Creative and better

New chipsets: x58, p55, p67, am3+ including UEFI bios support

"Multiseat" systems

Any atypical CPU like Via C2000 with appropriate chipsets

Motherboards with CD/DVD - JMicron controllers

As much as possible systems with SATA HDD connected with AHCI

SSD and RAID ( 1+0, 6 )

Any WIFI dongles and cards

3G modems - especially ZTE

Yota modems

SkyLink modems

Any mobile phones

Photo cameras which are not recognized as flash-memory (usually working with GPHOTO2)

Any printing machines - especially Win-printers

Interactive boards

Web-cameras Logitech - especially with Skype

Any Bluetooth gadgets - especially keyboards and mice

Test plans

1) Plan-report to be filled with Improver after preparing the data of the tests on preparation media, installation and loading the system and also work with virtual machines

1.1 Bootable DVD-ROM successfully recorded with Windows utility (please, specify)

1.2 Bootable DVD-ROM successfully recorded with Growisofs and other LINUX utilities (please, specify)

1.3 Bootable DVD-ROM successfully recorded with dd, (a command like: dd if = image.iso of = / dev / sd …)

2.1 When loading from the obtained DVD-ROM in the live mode all stages of the 1st launch work properly. (In case of errors, please, provide the full text of the errors.)

2.2 When you load in the LIVE  mode, the live user logs in automatically

2.3 When you boot in the LIVE mode there is no unnecessary data requests (please, specify what data is requested)

2.4 On each stage of booting there is an expressed splash screen. The themes of grub/syslinux splash, plymouth splash, ksplash correspond the system brand.

2.5 The algorithm Quit works properly

2.6 The algorithm Lock works properly

2.7 The algorithm Switch User works properly

2.8 The algorithm Sleep Mode works properly

2.9 The algorithm Reboot works properly: After you restart the system there should not be any changes to the hard drive of your PC. Especially check that the boot record of the hard disk is not demaged.

2.10 The algorithm Switch off' works properly: After you restart the system there should not be any changes to the hard drive of your PC. Especially check that the boot record of the hard disk is not demaged.

3 Burn a bootable image on flash device with a command like livecd-iso-to-disk / path / image.iso / dev / sd ... command type. The result should be a flash device with two folders DATA and SYSTEM. System should start successfully from this device.

4.1 When you load with the received USB-drive in live mode all stages of the 1st launch work properly.(In case of errors, please, provide the full text of the errors.)

4.2 When you load with the received USB-drive in live mode the live user logs in automatically

4.3 When you boot from the flash drive in the LIVE mode there is no unnecessary data requests (please, specify what data is requested)

4.4 On each stage of booting there is an expressed splash screen. The themes of grub/syslinux splash, plymouth splash, ksplash correspond the system brand.

4.5 After loading the live a mode from the written flash disk it is possible to save files in the root folder and in other folders (specify the restrictions).

4.6 Try to make some custom changes to the system: change the widgets, remove rpm-package (it is necessary to remove a package correctly, for example, the package kde-l10n-en_GB or any package with another option - nodeps. In a footnote you have to indicate that it is done and provide the output of rpm-qa | grep …)

4.7 Reboot: After you restart the system all the changes have to operate normally. (In case of unsaved changes you have to make a detailed report, as audited and the conclusion of cat / etc / sysconfig / draklive-install command)

4.8 Switch off: After you restart the system all the changes are present. (In case of unsaved changes, please, make a detailed report, indicate how did you check the changes, for example, provide the output of cat / etc / sysconfig / draklive-install command)

4.9 From the modified system on the flash-device from the Live mode install the system on the hard disk (draklive-install in the console) and make sure that the changes are saved and operate in the installed system (provide the output of rpm-qa | grep ... and / or screenshots).

5 Install the system from DVD-disk in 'Install System' mode. In case of any errors during the installation, make a photo picture of the screen then boot the live mode and run the installer from console with the command draklive-install-lock-storage. Try to reproduce the errors and provide the output in the console.

5.1 Graphical theme draklive-install accords with the system brand.

5.2 During installation there are no problems with recognising equipment and video modes (even if unsupported video card) 5.3 Network with its physical accessibility rises automatically and automatic update of packages runs.

5.4 At each step all the actions proposed in the form of a step perform properly (all the features work). Work out all the options of choice proposed by the installation wizard skiping / selecting various optional items such as removing unwanted software. Report any problems in detail.

5.5 After installation it is possible to load another operating system previously installed on the computer (specify the system).

6 Install the system from USB-flash-drive in 'Install System' mode. In case of any errors during the installation, make a photo picture of the screen then boot the live mode and run the installer from console with the command draklive-install-lock-storage. Try to reproduce the errors and provide the output in the console.

6.1 Graphical theme draklive-install accords with the system brand.

6.2 During installation there are no problems with recognising equipment and video modes (even if unsupported video card)

6.3 Network with its physical accessibility rises automatically and automatic update of packages runs

6.4 At each step all the actions proposed in the form of a step perform properly (all the features work). Work out all the options of choice proposed by the installation wizard skiping / selecting various optional items such as removing unwanted software. Report any problems in detail.

6.5 After installation it is possible to load another operating system previously installed on the computer (specify the system).

7.0.1 Choose the host system and install a free copy of Oracle VirtualBox virtual machine http://www.virtualbox.org/wiki/Downloads. Specify the selected host system.

7.0.2 Choose the host system and install a free copy of VMPlayer virtual machine http://downloads.vmware.com/d/info/desktop_downloads/vmware_player/3_0. Specify the selected host system.

7.0.3 Choose the host system and install a free copy of Parallels virtual machine. Specify the selected host system.

7.1.1 For Oracle VirtualBox virtual machine recommended confoguration is: from 1 GB RAM and from 8 GB HDD; minimal configuration is: from 512 MB RAM and from 6 GB HDD. Correct and/or add the specified parameters (configuration of processors, video memory etc.). At start choose 'LAUNCH SYSTEM'. The system should load fully. The network should connect automatically. Make screenshots from the host system sowing the process and result of booting the LIVE-mode in VirtualBox. Please, make the screenshots while the virtual machine is running not in full screen mode.

7.1.1.1 Test pre-installed vidgets on the taskbar including StackFolder. Make the screenshots from the host system.

7.1.1.2 Test the launch of all the applications from the Start Menu. Show the output of 'uname -a' from the root user in Console. Make the screenshots from the host system.

7.1.1.3 Perform some simple and representative operations like browsing a WEB site or any commands in the Console. Provide comments about speed, stability etc.

7.1.2 Restart the system and choose 'INSTALL SYSTEM'. Perform installation on Vbox HDD.

7.1.2.1 Make screenshots from the host system sowing the process and result of installing system on VirtualBox HDD. Please, make the screenshots while the virtual machine is running not in full screen mode.

7.1.2.2 Test pre-installed vidgets on the taskbar including StackFolder. Make the screenshots from the host system.

7.1.2.3 Test the launch of all the applications from the Start Menu. Show the output of 'uname -a' from the root user in Console. Make the screenshots from the host system.

7.1.2.4 Perform some simple and representative operations like browsing a WEB site or any commands in the Console. Provide comments about speed, stability etc.

7.2.1 For VMPlayer virtual machine recommended confoguration is: from 1 GB RAM and from 8 GB HDD; minimal configuration is: from 512 MB RAM and from 6 GB HDD. Correct and/or add the specified parameters (configuration of processors, video memory etc.). At start choose 'LAUNCH SYSTEM'. The system should load fully. The network should connect automatically. Meke screenshots from the host system sowing the process and result of booting the LIVE-mode in VMPlayer. Please, make the screenshots while the virtual machine is running not in full screen mode.

7.2.1.1 Test pre-installed vidgets on the taskbar including StackFolder. Make the screenshots from the host system. 7.2.1.2 Test the launch of all the applications from the Start Menu. Show the output of 'uname -a' from the root user in Console. Make the screenshots from the host system.

7.2.1.3 Perform some simple and representative operations like browsing a WEB site or any commands in the Console. Provide comments about speed, stability etc.

7.2.2 Restart the system and choose 'INSTALL SYSTEM'. Perform installation on VMPlayer HDD.

7.2.2.1 Make screenshots from the host system sowing the process and result of installing system on VirtualBox HDD. Please, make the screenshots while the virtual machine is running not in full screen mode.

7.2.2.2 Test pre-installed vidgets on the taskbar including StackFolder. Make the screenshots from the host system.

7.2.2.3 Test the launch of all the applications from the Start Menu. Show the output of 'uname -a' from the root user in Console. Make the screenshots from the host system.

7.2.2.4 Install VMTools. To start the installer use a command like perl ./vmware-install.pl Make the screenshots.

7.2.2.5 Perform some simple and representative operations like browsing a WEB site or any commands in the Console. Provide comments about speed, stability etc.

7.3.1 For Parallels virtual machine recommended confoguration is: from 1 GB RAM and from 8 GB HDD; minimal configuration is: from 512 MB RAM and from 6 GB HDD. Correct and/or add the specified parameters (configuration of processors, video memory etc.). At start choose 'LAUNCH SYSTEM'. The system should load fully. The network should connect automatically. Meke screenshots from the host system sowing the process and result of booting the LIVE-mode inParallels virtual machine. Please, make the screenshots while the virtual machine is running not in full screen mode.

7.3.1.1 Test pre-installed vidgets on the taskbar including StackFolder. Make the screenshots from the host system.

7.3.1.2 Test the launch of all the applications from the Start Menu. Show the output of 'uname -a' from the root user in Console. Make the screenshots from the host system.

7.3.1.3 Perform some simple and representative operations like browsing a WEB site or any commands in the Console. Provide comments about speed, stability etc.

7.3.2 Restart the system and choose 'INSTALL SYSTEM'. Perform installation on Parallels HDD.

7.3.2.1 Make screenshots from the host system sowing the process and result of installing system onParallels HDD. Please, make the screenshots while the virtual machine is running not in full screen mode.

7.3.2.2 Test pre-installed vidgets on the taskbar including StackFolder. Make the screenshots from the host system.

7.3.2.3 Test the launch of all the applications from the Start Menu. Show the output of 'uname -a' from the root user in Console. Make the screenshots from the host system.

7.3.2.4 Perform some simple and representative operations like browsing a WEB site or any commands in the Console. Provide comments about speed, stability etc.

2) Test plan for LIVE-DVD. To be performed in LIVE-DVD mode (LIVE-USB for netbooks).

1.1 The harware present in the testing computer defined and tuned correctly/ Plelease, use as much different hardware as possible according to the list of recommended hardware (please, see http://wiki.rosalab.ru/index.php/Improver_%28English_version%29 ) . Test connecting and/or tunning each device. Provide the data.

1.2 Network adjustments set and the connection established automatically when a network is present. If the connection is not established or is uncontrollable provide the output of 'ifconfig'.

2 The icons should be line up on the upper right edge on the desktop. Type of the desktop should be folder browsing. The size of the desktop icons 64x64 and signatures should not be circumcised.

3.1 After login the launch pannel (taskbar) appears on the bottom of the screen.

3.2 All the preinstalled widgets on the taskbar operate nominally (test all their functions and settings) and have no visible defects.

3.3 On the right side of the taskbar there are applets 'Stack Folder'. Click on the applet should open an appropriate folder.

3.3.1 Click on an object inside the stack folder should activate (open) the object.

3.3.2 Click on an icon on the right upper corner of the stack folder should open the folder in the file browser and the stack folder should automatically close.

3.3.3 The window of a stack folder is bigger or smaller depending on the number of the elements inside.

3.3.4 Grahpical effect of changing the folder performed smoothly without flicker.

3.3.5 Perform the following and check if there is no any problem of drawing: 1) Open a window of any GUI application on the full screen, for example, Dolphin. 2) Set a terminal window over it. 3) Open a Stack Folder window and go to a folder inside it containing a less number of elements so that size of the window would change. 4) There should not be any trace of the background of the window seen when it changes it's size.

4 There is the Start Menu applet on the left of the taskbar.

4.1 All items of the start menu function without error messages and other non-standard effects. All the applications open, perform their basic functions and close.

4.2 Test start menu editor (opens from the context menu on a right mouse button click on the start menu). Add / delete command.

4.3 TImeFrame allows to browse images and video files in a time scale depending on the time of the last modofication of the file. Copy some photo and video documents to the Home folder and it's subfolders. Then open the Start menu and go to the tab 'TimeFrame'.

4.3.1 In fields for display the data for the current month should be seen an animation showing that a search inquiry is processed. At occurrence of the first results animation will disappear. In the field over a time scale miniatures of the photos (up to 13) and a photo inscription should be displayed. In the field under a time scale video miniatures (up to 13) and video inscription should be displayed. Files with the big rating should have a bigger size.

4.3.2 To change an interval of search: click with a mouse on an interesting interval on a time scale. The toddler on a time scale will move on the chosen interval. Searching the new data will begin.

4.3.3 To change gradation of a time scale press button "D" or "M". The gradation of the scale will change on numbers, at a click on button "D", or for months, at a click on "M". Searching the new data will begin.

4.3.4 To change year/month: press the button «<» «>». The inscription designating year or year and month (depending on gradation of the scale) will change. Searching the new data will begin.

4.3.5 To see all elements (photo or video) for the set period click button "Photo" or "Video". The time scale will leave in the bottom part of the screen. In area for display the elements located one after another in the form of gallery will start to appear.

4.3.6 At prompt on a miniature there is an emerging help with name and the file rating.

4.3.7 At a click on an element the browser of images/videos will start.

4.4 Go to 'Welcome' tab (left one) and test the features there.

5.1 Elementary graphical theme enabled by default. Drawing of windows theme is dekorator. All the windows fitted withfolding, plowing and closing functions (in the upper right corner) and correctly perform these functions.

5.2 Open windows applications correspond to their minimized equivalents in in the taskbar. Right-click on the window panel activates the main window, and left click opens a context menu where all the items are processed correctly (test all their functions).

5.3 There is an application icon in the upper right corner of the application, left-click of the mouse button leads to the same contextual menu opening where all the items are processed correctly (test all their functions).

6 The keyboard layout is switched by pressing the specified key combination

7 All the elements of the context menus on the right-click on the desktop and on the elements on it function correctly (test the functions of all the items).

8 Internet Browser FireFox starts correctly by clicking the icon on the launch panel. The browser passes the test for basic functions and stability of work (installed plug-ins, stable play streaming audio and video and so on).

9 Is the variable PATH configured properly. Provide the output of env | grep PATH =

10 Test Plasma tools (except adding widgets).

11 With Dolphin and MC test the operations of copying and moving folders that contain many elements, including elements of the large size (more than gigabyte), as well as the presence in the target folder, a folder with a name identical to the copied folder, and contains multiple elements including identical elements that have a place in the copied folder. Test copying from disk to removable media devices and vice versa. Check automatic execution, what messages which are issued? Check whether there is a difference in the speed of copying the large files with Dolphin and MC. Attach protocol of the tests.

3) The Main test-plan. To be performed in the system installed on the hard disk.

1 There are no errors in the process of booting the system in safe mode. In the case of an error arising with downloading you have to attach to the problem the full text of the error (please, make a photo of the screen or retype the text), as well as the log-files of dmesg and x-org.

2 There are no errors in the process of booting the system in regular mode. In the case of an error arising with downloading you have to attach to the problem the full text of the error (please, make a photo of the screen or retype the text), as well as the log-files of dmesg and x-org.

3 On each stage of booting there is an expressed splash screen. The themes of grub/syslinux splash, plymouth splash, ksplash correspond the system brand.

5 Evaluate booting speed from your own point of view. How much time does it take to boot?

6 At first boot it is possible to create a user and login under his name.

7 There are no errors during user login.

8 Network adjustments set and the connection established automatically when a network is present. If the connection is not established or is uncontrollable provide the output of 'ifconfig'.

9 The harware present in the testing computer defined and tuned correctly/ Plelease, use as much different hardware as possible according to the list of recommended hardware (please, see http://wiki.rosalab.ru/index.php/Improver_%28English_version%29 ) . Test connecting and/or tunning each device. Provide the data.

10 The icons should be line up on the upper right edge on the desktop. Type of the desktop should be folder browsing. The size of the desktop icons 64x64 and signatures should not be circumcised.

11.1 After login the launch pannel (taskbar) appears on the bottom of the screen.

11.2 All the preinstalled widgets on the taskbar operate nominally (test all their functions and settings) and have no visible defects.

11.3 On the right side of the taskbar there are applets 'Stack Folder'. Click on the applet should open an appropriate folder.

11.3.1 Click on an object inside the stack folder should activate (open) the object.

11.3.2 Click on an icon on the right upper corner of the stack folder should open the folder in the file browser and the stack folder should automatically close.

11.3.3 The window of a stack folder is bigger or smaller depending on the number of the elements inside.

11.3.4 Grahpical effect of changing the folder performed smoothly without flicker.

11.3.5 Perform the following and check if there is no any problem of drawing: 1) Open a window of any GUI application on the full screen, for example, Dolphin. 2) Set a terminal window over it. 3) Open a Stack Folder window and go to a folder inside it containing a less number of elements so that size of the window would change. 4) There should not be any trace of the background of the window seen when it changes it's size.

12 There is the Start Menu applet on the left of the taskbar.

12.1 All items of the start menu function without error messages and other non-standard effects. All the applications open, perform their basic functions and close.

12.2 Test start menu editor (opens from the context menu on a right mouse button click on the start menu). Add / delete command.

12.3 TImeFrame allows to browse images and video files in a time scale depending on the time of the last modofication of the file. Copy some photo and video documents to the Home folder and it's subfolders. Then open the Start menu and go to the tab 'TimeFrame'.

12.3.1 In fields for display the data for the current month should be seen an animation showing that a search inquiry is processed. At occurrence of the first results animation will disappear. In the field over a time scale miniatures of the photos (up to 13) and a photo inscription should be displayed. In the field under a time scale video miniatures (up to 13) and video inscription should be displayed. Files with the big rating should have a bigger size.

12.3.2 To change an interval of search: click with a mouse on an interesting interval on a time scale. The toddler on a time scale will move on the chosen interval. Searching the new data will begin.

12.3.3 To change gradation of a time scale press button "D" or "M". The gradation of the scale will change on numbers, at a click on button "D", or for months, at a click on "M". Searching the new data will begin.

12.3.4 To change year/month: press the button «<» «>». The inscription designating year or year and month (depending on gradation of the scale) will change. Searching the new data will begin.

12.3.5 To see all elements (photo or video) for the set period click button "Photo" or "Video". The time scale will leave in the bottom part of the screen. In area for display the elements located one after another in the form of gallery will start to appear.

12.3.6 At prompt on a miniature there is an emerging help with name and the file rating.

12.3.7 At a click on an element the browser of images/videos will start.

12.4 Go to 'Welcome' tab (left one) and test the features there.

13.1 Elementary graphical theme enabled by default. Drawing of windows theme is dekorator. All the windows fitted withfolding, plowing and closing functions (in the upper right corner) and correctly perform these functions.

13.2 Open windows applications correspond to their minimized equivalents in in the taskbar. Right-click on the window panel activates the main window, and left click opens a context menu where all the items are processed correctly (test all their functions).

13.3 There is an application icon in the upper right corner of the application, left-click of the mouse button leads to the same contextual menu opening where all the items are processed correctly (test all their functions).

14 The keyboard layout is switched by pressing the specified key combination

15 All the elements of the context menus on the right-click on the desktop and on the elements on it function correctly (test the functions of all the items).

16.1 Internet Browser FireFox starts correctly by clicking the icon on the launch panel. The browser passes the test for basic functions and stability of work (installed plug-ins, stable play streaming audio and video and so on).

16.2 Test e-mail client Tunderbird (create boxes POP and IMAP, test the common functions).

17 Is the variable PATH configured properly. Provide the output of env | grep PATH =

18 Test Plasma tools (except adding widgets).

19 Test adding all the available widgets on the desktop (with no raids of text on icons, the icons on the text and icons on the icons, check the functions and configurations of the widgets).

20 Test adding all the available widgets on the launch panel (with no raids of text on icons, the icons on the text and icons on the icons, check the functions and configurations of the widgets).

21 With Dolphin and MC test the operations of copying and moving folders that contain many elements, including elements of the large size (more than gigabyte), as well as the presence in the target folder, a folder with a name identical to the copied folder, and contains multiple elements including identical elements that have a place in the copied folder. Test copying from disk to removable media devices and vice versa. Check automatic execution, what messages which are issued? Check whether there is a difference in the speed of copying the large files with Dolphin and MC. Attach protocol of the tests.

22 By means of the system monitor provide the data about memory used by system: immediately after loading, and then in an hour or two after loading without log off the system.