An Insight View of Kernel Visual Debugger in System Boot up
IInternational Journal of Computer Science & Information Technology (IJCSIT) Vol 4, No 5, October 2012DOI : 10.5121/ijcsit.2012.4510 115
An Insight View of Kernel Visual Debugger inSystem Boot up
Mohamed Farag
Department of Computer Science, Maharishi University of Management, USA [email protected]
ABSTRACT
For many years, developers could not figure out the mystery of OS kernels. The main source of this mystery is the interaction between operating systems and hardware while system’s boot up and kernel initialization.
In addition, many operating system kernels differ in their behavior toward many situations. For instance,kernels act differently in racing conditions, kernel initialization and process scheduling. For suchoperations, kernel debuggers were designed to help in tracing kernel behavior and solving many kernelbugs. The importance of kernel debuggers is not limited to kernel code tracing but also, they can be used inverification and performance comparisons. However, developers had to be aware of debugger commandsthus introducing some difficulties to non-expert programmers. Later, several visual kernel debuggers werepresented to make it easier for programmers to trace their kernel code and analyze kernel behavior.Nowadays, several kernel debuggers exist for solving this mystery but only very few support line-by-linedebugging at run-time. In this paper, a generic approach for operating system source code debugging ingraphical mode with line-by-line tracing support is proposed. In the context of this approach, system bootup and evaluation of two operating system schedulers from several points of views will be discussed.
KEYWORDS
Debugging; Kernel; Linux; FreeBSD; Qemu
1. INTRODUCTION
A kernel is a central component of an operating system. It acts as an interface between the userapplications and the hardware. The sole aim of the kernel is to manage the communicationbetween the software (user level applications) and the hardware (CPU, disk memory etc). Themain tasks of the kernel are: Process Management, Device Management, Memory Management,Interrupt Handling, I/O Communication and File System. Debugging is a methodical process offinding and reducing the number of bugs, or defects, in a computer program or a piece ofelectronic hardware, thus making it behaves as expected. As software and electronic systems havebecome generally more complex, the various common debugging techniques have expanded withmore methods to detect anomalies, assess impact, and schedule software patches or full updates toa system. An OS kernel debugger is a debugger presented insome kernels for ease debugging andkernel development. There are several techniques to implement kernel debugging such as printingdebugging, remote debugging, post-mortem debugging[1], delta debugging[2] and SaffSqueez[3]. Remote debugging is the process of debugging a program running on a systemdifferent than the debugger machine. In remote debugging, a debugger connects to a remote nternational Journal of Computer Science & Information Technology (IJCSIT) Vol 4, No 5, October 2012116 system over a network. Once connected, the debugger can control the execution of the programon the remote system and retrieve information about its execution. Remote debugging followsclient/server architecture with TCP/IP or COM for communication between machines.The role ofthe client machine is to debug the code existing on the server but server machine ’s application should have enabled debugging support to permit another machine to debug its code. In addition,a copy of the application running on the remote hardware should be kept in the client side to beable to view it. This copy should have the same debug information as the original. If the remoteapplication is using another target(i.e. different hardware architecture), client debugger shouldalso support target ’s platform.
2. RELATED WORK
Several attempts were made for kernel debugging. Several visual kernel debuggers were designedfor this purpose such as DDD and WinDbg. GNU DDD is a graphical front-end for command-line debuggers such as GDB, DBX, WDB, Ladebug, JDB, XDB, the Perl debugger, the bashdebugger bashdb, the GNU Make debugger remake, or the Python debugger pydb. Besides``usual'' front-end features such as viewing source texts, DDD has become famous through itsinteractive graphical data display, where data structures are displayed as graphs. WinDbg is amulti- purposed debugger for Microsoft Windows, distributed on the web by Microsoft. It can beused to debug user mode applications, drivers, and the operating system itself in kernel mode. Itis a GUI application, but has little in common with the more well-known, but lesspowerful, Visual Studio Debugger[5].
3. PROBLEM DESCRIPTION & SUGGESTED SOLUTION
Existing visual kernel debuggers have some problems. Many visual debuggers supporting kerneldebugging are front end views for kernel debuggers so some of them are not fully compatiblewith the debugger itself [6] . For example, DDD debugger which is a front end for GNU GDB but it doesn’t work with old editions of
GDB. In addition, some debuggers are platform dependant suchas WinDbg which is Windows-specific debugger. On the other hand, some debuggers don’t provide assembly level debug along with C debugging rather than visualizing the code [4] .Furthermore, debuggers should use at least two machines to debug the code remotely hencemaking it costly to test such machines. According to the mentioned in the last paragraph, there’s asolution satisfying the following criteria:1. Platform-Independent.2. Portable.3. Supporting the debug of different programming languages at the same run (C andAssembly).4. High compatibility between the front end and the debugger itself.5. User friendly.6. Low Cost.7. Open Source with appropriate license (GPL and EPL).These criteria can be satisfied using tools reconfiguration and there’s no need to develop new tool or write code for that.The solution we are providing is an enhanced version ofremotedebugging.However, Fig 1 shows typical remote debugging environment while the topologyfollowed in this paper is the one provided in Fig 2. nternational Journal of Computer Science & Information Technology (IJCSIT) Vol 4, No 5, October 2012117Fig 1 Remote Debugging TopologyFig 2 Enhanced Remote Debugging Topology
In the suggested approach, different metrics were in consideration according to the followingpoints: OSQemuGDBServer
Eclipse
HOSTLoopback Interface APP
GDB SERVER
OSDDDGDBOS
TARGET
HOST TCP/IP, COM nternational Journal of Computer Science & Information Technology (IJCSIT) Vol 4, No 5, October 2012118
1. Take the advantage of Eclipse CDT and Qemu as portable, platform independent, OpenSource and reliable tools supporting this technique under several platforms.2. Eclipse support different debuggers and can combine many of them under the same run.3. Eclipse has its own GDB plugin which guarantees high compatibility.4. Only one machine is used for lower cost.In suggested approach, Eclipse CDT, which supports C & assembly languages, connects to GDBServer for debugging. GDB Server used TCP/IP protocol stack with tricky loopback interface toconnect to Qemu. In separate thread of execution, Qemu is pointed to target kernel image andcontrolled by GDB Server commands to trace kernel execution.
4. METHODOLOGY
The main idea behind this implementation is to command Qemu to point to the kernel bootinitializer. Behind the scene, boot initializer will initialize the kernel and start it. In the same time,GDB Server debugger is tracking the kernel image itself to provide connection point betweenEclipse and Qemu through recognizing line of code being executed and commanding Qemu.
Let’s consider linux kernel debugging t o clarify the state of the art in this idea,. If linux kernel iscompiled, kernel boot initializer will be stored in arch/ machine_arch /boot/bzimage under thekernel compilation directory. In addition, the kernel image will be co-located in the parent kerneldirectory under the name vmlinux. The following figure describes this scenario in detail. Pleaseconsider the numbers above the arrows to indicate the order of steps. nternational Journal of Computer Science & Information Technology (IJCSIT) Vol 4, No 5, October 2012119Fig 3 Run-time Debugging Communication Diagram
The following figure shows an eclipse screen shotat the scheduler initialization function“ sched_init() ” . Fig 4 Eclipse Debugger screenshot
Emulator(Qemu) Boot Initializer ( bzimage)Kernel Image(vmlinux)Debugger(Eclipse/CDT)
1. Emulate 3. Start2. Debug4. Run-timesource codeviewing&outputshown inQemu
Linux Kernel Visual Debugging nternational Journal of Computer Science & Information Technology (IJCSIT) Vol 4, No 5, October 2012120
5. CASE STUDY (FREEBSD OS BOOT)
There’s no doubt that system scheduler is the most important part of operating system due to itsperformance effect according to preemption or non-preemption rules. One of the great debugging mysteries is the system’s boot -up process which describes how kernel could be initialized andturned on to an active state. Nowadays, debuggers provide huge capabilities such as performancemeasurements and code verification. One of the approaches used for performance measurementsis Line Of Code (LOC) approach. Usually, number of line of code will be multiplied by theaverage time to execute one command (Typically, 10ns) resulting the total average time. Next,this technique was used to analyze FreeBSD Boot-up and to compare the performance of twocommon FreeBSD schedulers. The first one is ULE Scheduler and the second one is 4.3 BSDScheduler. Notice that another platform is used in this case study to show how compatible is thisapproach with other platforms. FreeBSD 8.0 Machine on 32-bit is used for this experiment. Thefirst output of the program was control flow of FreeBSD booting process.
Turn on PCRead the value 0xfffffff0 from instruction pointerregisterHardware translates the value to read BIOSMemory Block
Jump to BIOS’s POST routines
POST performs different checksOne of the POST instructions is INT0x19 instruction which reads 512 bytes from thefirst sector of boot device into the memory ataddress 0x7c00
International Journal of Computer Science & Information Technology (IJCSIT) Vol 4, No 5, October 2012120
5. CASE STUDY (FREEBSD OS BOOT)
There’s no doubt that system scheduler is the most important part of operating system due to itsperformance effect according to preemption or non-preemption rules. One of the great debugging mysteries is the system’s boot -up process which describes how kernel could be initialized andturned on to an active state. Nowadays, debuggers provide huge capabilities such as performancemeasurements and code verification. One of the approaches used for performance measurementsis Line Of Code (LOC) approach. Usually, number of line of code will be multiplied by theaverage time to execute one command (Typically, 10ns) resulting the total average time. Next,this technique was used to analyze FreeBSD Boot-up and to compare the performance of twocommon FreeBSD schedulers. The first one is ULE Scheduler and the second one is 4.3 BSDScheduler. Notice that another platform is used in this case study to show how compatible is thisapproach with other platforms. FreeBSD 8.0 Machine on 32-bit is used for this experiment. Thefirst output of the program was control flow of FreeBSD booting process.
Turn on PCRead the value 0xfffffff0 from instruction pointerregisterHardware translates the value to read BIOSMemory Block
Jump to BIOS’s POST routines
POST performs different checksOne of the POST instructions is INT0x19 instruction which reads 512 bytes from thefirst sector of boot device into the memory ataddress 0x7c00
International Journal of Computer Science & Information Technology (IJCSIT) Vol 4, No 5, October 2012120
5. CASE STUDY (FREEBSD OS BOOT)
There’s no doubt that system scheduler is the most important part of operating system due to itsperformance effect according to preemption or non-preemption rules. One of the great debugging mysteries is the system’s boot -up process which describes how kernel could be initialized andturned on to an active state. Nowadays, debuggers provide huge capabilities such as performancemeasurements and code verification. One of the approaches used for performance measurementsis Line Of Code (LOC) approach. Usually, number of line of code will be multiplied by theaverage time to execute one command (Typically, 10ns) resulting the total average time. Next,this technique was used to analyze FreeBSD Boot-up and to compare the performance of twocommon FreeBSD schedulers. The first one is ULE Scheduler and the second one is 4.3 BSDScheduler. Notice that another platform is used in this case study to show how compatible is thisapproach with other platforms. FreeBSD 8.0 Machine on 32-bit is used for this experiment. Thefirst output of the program was control flow of FreeBSD booting process. nternational Journal of Computer Science & Information Technology (IJCSIT) Vol 4, No 5, October 2012121Fig 5 Control Flow for FreeBSD 8.0
An important advantage of kernel source level debugging is to detect the control flow diagram.On the other hand, it can be used to specify function responsibilities and how they relate to theoverall execution. Next, all the boot functions are described in the following steps:1. boot0
Location sys/boot/i386/boot0/boot0.S
Memory
INT 0x19 instruction loads an MBR, i.e. the boot0 content, into thememory at address 0x7c00
MBR Structure
Starting from 0x1be, called the partition table,
It has 4 records of16 bytes each, called partition record
Partition RecordStructure • • • • File Location sys/boot/i386/boot2/boot2.c
Functionality • It scans the harddisk. • knowing about the filesystem structure • finds the file /boot/loader • reads it into memory using a BIOS service • passes the execution to the loader's entry point. • boot2 prompts for user input so the loader can be bootedfrom different disk, unit, slice and partition • passes the execution to BTXTable 2.0 boot2 properties boot0boot1boot2loaderinit International Journal of Computer Science & Information Technology (IJCSIT) Vol 4, No 5, October 2012121Fig 5 Control Flow for FreeBSD 8.0
An important advantage of kernel source level debugging is to detect the control flow diagram.On the other hand, it can be used to specify function responsibilities and how they relate to theoverall execution. Next, all the boot functions are described in the following steps:1. boot0
Location sys/boot/i386/boot0/boot0.S
Memory
INT 0x19 instruction loads an MBR, i.e. the boot0 content, into thememory at address 0x7c00
MBR Structure
Starting from 0x1be, called the partition table,
It has 4 records of16 bytes each, called partition record
Partition RecordStructure • • • • File Location sys/boot/i386/boot2/boot2.c
Functionality • It scans the harddisk. • knowing about the filesystem structure • finds the file /boot/loader • reads it into memory using a BIOS service • passes the execution to the loader's entry point. • boot2 prompts for user input so the loader can be bootedfrom different disk, unit, slice and partition • passes the execution to BTXTable 2.0 boot2 properties International Journal of Computer Science & Information Technology (IJCSIT) Vol 4, No 5, October 2012121Fig 5 Control Flow for FreeBSD 8.0
An important advantage of kernel source level debugging is to detect the control flow diagram.On the other hand, it can be used to specify function responsibilities and how they relate to theoverall execution. Next, all the boot functions are described in the following steps:1. boot0
Location sys/boot/i386/boot0/boot0.S
Memory
INT 0x19 instruction loads an MBR, i.e. the boot0 content, into thememory at address 0x7c00
MBR Structure
Starting from 0x1be, called the partition table,
It has 4 records of16 bytes each, called partition record
Partition RecordStructure • • • • File Location sys/boot/i386/boot2/boot2.c
Functionality • It scans the harddisk. • knowing about the filesystem structure • finds the file /boot/loader • reads it into memory using a BIOS service • passes the execution to the loader's entry point. • boot2 prompts for user input so the loader can be bootedfrom different disk, unit, slice and partition • passes the execution to BTXTable 2.0 boot2 properties nternational Journal of Computer Science & Information Technology (IJCSIT) Vol 4, No 5, October 2012122
3. Loader
File Location sys/boot/i386/boot/loader
Notes • kernel bootstrapping final stage • When the kernel is loaded into memory, it is being calledby the loader
Functionality • The main task for the loader is to boot the kernel. • It provides a scripting language that can be used toautomate tasks, do pre-configuration or assist in recoveryprocedures.
Table 3.0 Loader properties
4. init386()
File Location sys/i386/i386/machdep.c
Functionality • Initialize the kernel tunable parameters, passed from thebootstrapping program. • Prepare the GDT. • Prepare the IDT. • Initialize the system console. • Initialize the DDB, if it is compiled into kernel. • Initialize the TSS. • Prepare the LDT. • Set up proc0's pcb
Table 4.0 init386 Properties
After complete analysis of system component responsibilities, this approach was used in moreeffective way to compare the execution time of two common schedulers in FreeBSD. Thefollowing results show that 4.3BSD scheduler is much more efficient than ULE scheduler throughthe installation of Apache port. Time units are in seconds but the most important are thedifferences in the results. On Pentium-4 Machine, real time statistics are presented in thefollowing table:
ConcurrentProcesses ULE ULEStddev BSD BSDStddev faster2 2371.9 1.212 2346.29 1.89 4BSD4 2007.8 2.58 1999 0.68 4BSDTable 5.0 Real Time Statistics for Schedulers
User Time statistics are presented in the following table:
ConcurrentProcesses ULEUser ULEStddev BSDUser BSDStddev faster2 2251.9 5.3 2221.6 2.1 4BSD4 2499.9 2.74 2416.13 2.9 4BSDTable 6.0 User Time Statisticsnternational Journal of Computer Science & Information Technology (IJCSIT) Vol 4, No 5, October 2012123
System Time statistics are calculated in the following table:
ConcurrentProcesses ULESystem ULEStddev BSDSystem BSDStddev Faster2 434.9 2.05 408.7 1.28 4BSD4 499.2 1.65 465.16 2.228 4BSDTable 7.0 System Time Statistics for Schedulers
Now, it looks fairly easy to update operating system source code because it’s possible to check ifthe source code is acting as it should or not.
6. FUTURE WORK
This paper is the first step of OS bug detection system. The supposed plan starts by identifying setof possible behaviors for specific components of operating system thus checking updated codeagainst possible behaviors thus fixing the resulted bugs.
7. CONCLUSIONS
This paper introduced fairly easy technique to debug the operating system from both correctnessand effectiveness point of views according to some similarities in the behavior of differentoperating systems.
8. ACRONYMS • OS: Operating System • ddd: Data Display Debugger • Windbg: Windows Debugger • GDB: GNU Project Debugger • Qemu: Quick EMUlator. • Stddev: Standard Deviation • BSD: Berkeley Software Distribution • ULE: A FreeBSD Task Scheduler • GPL: GNU General Public License. • EPL: ECLIPSE PUBLIC LICENSE
9. ACKNOWLEDGEMENT
I thank Bruce Lester, PhD at Maharishi University of Management and Wail Al-Kilani, AssociateProfessor at Ain-Shams University for their support. I have learned a lot from Bruce when I gotthe opportunity to be teaching assistant in his parallel programming class nternational Journal of Computer Science & Information Technology (IJCSIT) Vol 4, No 5, October 2012124
10. REFERENCES [1] Matthew A. Telles, and Yuan Hsieh, "The Science of Debugging," The Coriolis Group, ISBN 1-57610-917-8, 2001.[2] Dmitry Vostokov, "Memory Dump Analysis Anthology," Volume 1, OpenTask, ISBN 978-0-9558328-0-2, 2008.[3] Andreas Zeller, "Why Programs Fail: A Guide to Systematic Debugging," Morgan Kaufmann, ISBN1-55860-866-4, 2005.[4] Ray Kinsella, "Profiling and Debugging the FreeBSD* Kernel," Document Number: 321772-001,2009.[5] S.David Ribo, and Clearwater, FL, "How to use the Data Step Debugger," Paper 52-25, (available for download from the author’s Web site: HTTP://WWW.JADETEK.COM/JADETECH.HTM) [6] Norman Matloff, The Art of Debugging with Gdb, DDD, and Eclipse, No Starch Press, ISBN:9781593271749, 2008
Author
Mohamed Farag is post graduate student in Maharishi University of Management in USA. In the first sixmonths of 2012, Mohamed worked as teaching assistant in Maharishi University of Management. He alsoworked as instructor in Ain Shams University in Egypt during the year 2011. In 2010, Mohamed receivedGoogle Summer of Code award and was honored by the scientific community in Menoufia University inEgypt.
In addition, Mohamed received “The Best Programming Project” award in Egyptian Universities
Summit in 2010 and the same award in 2011. Mohamed has been an active contributor in FreeBSDcommunity since May, 2010 and has led ArabBSD project since June, 2011. In 2012, Mohamed wasselected to join the Institute for Computer Sciences, Social Informatics and TelecommunicationsEngineering “ ICST ” , International Association of Computer Science and Information Technology “ IACSIT ” , Computer Science Teachers Association “ CSTA ACM ” and Academy & Industry ResearchCollaboration Center “ AIRCC ” . In September 2012, Mohamed was selected to be reviewer forInternational Journa l of Computer Science and Information Technology “IJCSIT” and program committeel of Computer Science and Information Technology “IJCSIT” and program committee