Published on Nov 30, 2023
In this project, the main part is, how can we implement video streaming on every android platform devices and how can we make a specified “tablet¨ devices that we want? In this project, we want to design a device which is designed on embed Android platform based on Linux Kernel to transmit real time video signals to any Android devices. To make it more interesting, we try to embed this system into a simple motor car and it provides the user playing the car with the first-person view likes a driving a car in the real world. For the performance, the “tablet” device runs smoothly and also video streaming is well performed in the good quality of network or high speed CPU environment. To deal with this issue, we have tried modified the program and its performance is finally improved. Apart from that, the motor car can be controlled smoothly by the user without any problems. This project may be useful for the people who want to study about video conferencing and tablet hardware design since we will explain the rationale and procedure behind of these two parts.
a) To study designing hardware techniques such as fabrication and soldering
b) To study drawing Schematic Diagram and routing PCB layout for hardware design by using Altium DXP
c) To design a module including 2 motor drivers in order to drive the car movement
d) To design a module including 6 motor drivers and one multiplexer in order to drive the robot arm
e) To study the reference design for the Arm Controlled Board of REAL6410
f) To redesign and make a specified functional device that we want based on the reference
g) To study the architecture of the core board E210COREV2
h) To design an Arm Controlled Board which is based on the core board E210COREV2 and the reference of Arm Controlled Board of REAL6410
Software part
a) To study Camera and Wi-Fi drivers which are provided by the sellers
b) To study the Linux Kernel, Android Operating System and U-Boot which are the open sources from the Internet and reference design (Real 6410)
c) To modify and transplant Wi-Fi and Camera drivers in the Linux Kernel
d) To port the Android Operating System, Linux Kernel and U-Boot to the own designed Arm Controlled Board by the reference REAL6410 and the own designed Arm Controlled Board based on the core board E210COREV2
e) To write and modify C-program for LPC1768 to control the movement of the car’s motors, servo motor and robot arm
f) To write an Android apps for the Arm Controlled Board which is embedded on the car including the following functions:
1. Can get the capture from the camera and real-time stream the video to client’s android mobile device which connected to the car
2. Can record and stream the video simultaneously and the recorded video is stored in the SD card of the EVB
3. Can receive and distinguish the command sent by client and send the command to LPC1768 through UART port of the EVB
g) To write an Android apps for the Android mobile device including the following functions:
1. Can receive the streamed video and display the image on the screen of Android mobile device in real-time
2. Can send the controlling command to the EVB which is embedded on the car through Wi-Fi or 3G
E6COREV3 is smoothly passed through the EMC testing. It is added JTAG interface on the basis of the E6COREV1, the R&D of E6COREV3 is strongly supported by Samsung China, WOLFSON (Shanghai), SanDisk (Shenzhen) and MPS (China) that support component selection and testing. Apart from that, it is worthy to mention that the Top layer and Bottom layer alignment of E6COREV3 is using 8 layers buried blind design. Therefore, it effectively prevents the electromagnetic radiation and electromagnetic interference. It is fully tested in the industrial environments.
The specification of E6COREV3:
1. ARM-11 Kernel
2. CPU Clock Rate 667MHZ
3. 256M Byte mDDR
4. 1GByte MLC NAND FLASH
5. WM9713 (WOLFSON Official Product), support mobile audio design project
6. iNAND (support 16GB SanDisk)
It has different ports in one reference EVB to developers for basic testing or developing. Now I am going to introduce that the ports we may use in this project and the ports are:
1. Mini-USB OTG port
2. USB Host 1.1 port
3. Two RS-232 ports
4. SD Card port
5. Two TTL UART ports
6. Camera port
7. SPI port
8. Embedded MIC
9. SDIO WIFI port
10. 24-bit RGB LCD port
Furthermore, it has provided different modules for developers for different specified testing or developing. Now I am going to introduce the module we may use in this project and the modules are:
1. 3G module
2. WIFI module
3. Camera module
4. 7 inch Touch Screen
The NXP LPC1768 which is founded by Philips is an ARM 32-bit Cortex-M3 Microcontroller with MPU, CPU clock up to 100MHz, 512kB onchip Flash ROM with enhanced Flash Memory Accelerator, In-System Programming (ISP) and In-Application Programming (IAP), 64kB RAM, Nested Vectored Interrupt Controller, Eight channel General purpose DMA controller, AHB Matrix, APB, Ethernet 10/100 MAC with RMII interface and dedicated DMA, USB 2.0 full-speed Device controller and Host/OTG controller with DMA, CAN 2.0B with two channels, Four UARTs, one with full Modem interface, Three I2C serial interfaces, Three SPI/SSP serial interfaces, I2S interface, General purpose I/O pins, 12-bit ADC with 8 channels, 10-bit DAC, Four 32-bit Timers with capture/compare, Standard PWM Timer block, Motor control PWM for three-phase Motor control, Quadrature Encoder, Watchdog Timer, Real Time Clock with optional Battery backup, System Tick Timer, Repetitive Interrupt Timer, Brown-out detect circuit, Power-On Reset, Power Management Unit, Wakeup Interrupt Controller, Crystal oscillator, 4MHz internal RC oscillator, PLL, JTAG and Serial Wire Debug/Trace Port with ETM.
Figure 3.1 shows a diagram about the system connection. First, Android Camera Car (Arm Controlled Board) is connected to the stable Wi-Fi network by a user and it will show the IP of the EVB on the screen. Afterward, when the user wants to control the car remotely, it is required to connect to a stable Wi-Fi network first, and then turn on our designed Android apps. After, the user enters the IP of the EVB and presses the connect button, the whole system will be connected and comes to a stable state. Simply pressing corresponding buttons, actions such as taking photos, recording video, controlling the servo (camera view) and turning on/off LED lights can be accomplished by the user.
Typically, the functions of the application in the Arm Controlled Board are:
1. Receiving controlling command from the client’s side;
2. Sending the video packet to the client’s side;
3. Determining and sending the command message to LPC1768 in order to control the movement of motors and servo and lighting LEDs;
4. Displaying the current command that the client’s want;
5. Displaying the IP of Arm Controlled Board;
6. Determine the IP of the client’s so that make the connection simpler; and
7. Recording the video in the SD card if the client’s want.
Theoretically, functions 2 and 6 (originally Jason is responsible for them) work when they run independently. However, when I tried to merge them into an application, there are conflicts between them. After my modification, the application now can run for many times as much as user want. For example, in my program, a thread is created for video streaming. However, when the user wants to record and stream the video simultaneously, the program needs to close the original thread and then open another thread for the simultaneous streaming and recording. In another example, if the user wants to exit the application and wants to notify the Car stop streaming video to the user. The question is how to achieve the closing of the original thread.
The functions of the application in the Android Mobile Devices are in the followings:
1. Sending controlling command such as controlling the movement of the car to the server’s side;
2. Receiving the video packet from the server’s side;
3. Decoding and displaying the video data on the screen;
4. Displaying the current command that the client’s want;
5. Sending the command of the car’s movement by using Motion Sensing;
6. Capturing and storing the picture from the client’s side; and
7. Improving the quality of the video streaming in poor CPU performance.
Theoretically, Jason is responsible for part 2, part 3 and part 6 and Chris is responsible for part 5 and it can be worked when they run independently. However, when I tried to merge them into an application, there are conflicts between them. After my modification, the application now can run, and furthermore it can accept users’ various connection requests. Apart from that, the video streaming is related to the performance of the CPU since it requires software decoding. That means the better performance the CPU has, the better quality of video streaming that the client can be received. If the performance of CPU is poor, some errors such as green or grey areas may display on the screen. In order to deal with this problem, I tried to create two threads to receive the video packets. However, the two threads cannot receive the video packets simultaneously. If they receive the packets at the same time, the frames will be received in a wrong arrangement. Therefore, “semaphore” is used to solve this problem.
Apart from that, in order to make the control more user-friendly, for example, controlling the movement of the car, I used onKeyListener but not the onClickListener since the former makes the control of the car more convenient. Furthermore, after pressing any button on the screen, the devices will vibrate in order to alarm users: “You have pressed the button successfully.” The purpose of those modifications is to make the control panel more user-friendly.
Figure 4.38 is a diagram showing the structure of the program in tree form. The main class of the program is FfmpegdecodeActivity{}. It is responsible for the command sending. Once the user enter the IP of Arm Controlled Board and click connect button, the function – PlayVideo in subclass Vview will start a new thread. The duty of this function is to receive and decode the video packets to display on the screen of client’s Android mobile devices. Another duty of this class can help the clients so capture the current streamed image when they want. Apart from that, SplashScreen{} is to show the opening and welcoming screen to the user. VerticalSeekBar{} is used to make the seek bar placed vertically since Android do not provide official vertical seek bar, we need to make it by ourselves.
Android Camera Car
In this project, there are many tasks for developing this product.
• The Schematic of Arm Controlled Board with ARM11
• The Schematic of Arm Controlled Board with Cortex-A8
• The PCB layout of the adaptor board with Cortex-A8
• Modifying the Camera Driver
• Integrating and modifying an Android Application of Arm Controlled Board
• Integrating and modifying an Android Application in Android Mobile Device
• Writing a program for controlling the motor driver
[1] Description of E6COREV3. Available from: http://www.realarm.cn/pic/?77_481.html
[2] Description of Arm Controlled Board for E6COREV3. Available from: http://www.realarm.cn/pic/?76_451.html
[3] Description of E210COREV2.Available from: http://www.realarm.cn/pic/?79_461.html
[4] Product information of LPC178. Available from: http://www.keil.com/dd/chip/4868.htm
[5] Soldering - Wikipedia, the free encyclopedia. Available from: http://en.wikipedia.org/wiki/Soldering
