Traffic Signs in the Wild: Highlights from the IEEE Video and Image Processing Cup 2017 Student Competition [SP Competitions]
CCitation
D. Temel and G. AlRegib, ”Traffic Signs in the Wild: Highlights from the IEEE Video and Image ProcessingCup 2017 Student Competition [SP Competitions],” in IEEE Signal Processing Magazine, vol. 35, no. 2, pp.154-161, March 2018.
DOI https://doi.org/10.1109/MSP.2017.2783449
Review
Date of publication: 7 March 2018
VIP https://ghassanalregib.com/vip-cup/
Bib @ARTICLE { Temel2018 SPM,author= { D. Temel and G. AlRegib } ,journal= { IEEE Signal Processing Magazine } ,title= { Traffic Signs in the Wild: Highlights from the IEEE Video and Image Processing Cup 2017 StudentCompetition [SP Competitions] } ,year= { } ,volume= { } ,number= { } ,pages= { } ,keywords= { traffic engineering computing;video signal processing;IEEE Video and Image Processing Cup 2017Student Competition;traffic signs } ,doi= { } ,ISSN= { } ,month= { March } , } Copyright ©2018 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for allother uses, in any current or future media, including reprinting/republishing this material for advertising orpromotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse ofany copyrighted component of this work in other works.
Contact [email protected] https://ghassanalregib.com/[email protected] http://cantemel.com/ a r X i v : . [ c s . C V ] N ov Traffic Signs in the Wild: Highlights from the IEEEVideo and Image Processing Cup 2017 StudentCompetition [SP Competitions]
Dogancan Temel and Ghassan AlRegib
As we witness the fourth industrial revolution, sev-eral aspects of our daily lives will soon be impactedbeyond recognition. The list includes healthcare, edu-cation, security, transportation, warfare, and entertain-ment. Transportation, in particular, is undergoing a setof disruptive technologies including electrical vehicles(EV) and autonomous vehicles (AV). Although AVshave witnessed a revolution in many aspects over thepast twenty years, deploying AVs in the wild remainsto be a challenge. One of the basic features of AVs is tounderstand the surroundings and interpret sensed data.This requires the deployment of recognition algorithmsthat are expected to operate under all conditions. Oneof the most researched recognition applications in theliterature is traffic sign recognition (TSR). Nevertheless,testing TSR algorithms under challenging conditionshas been lagging for a number of reasons. One majorfactor is the limitation of existing datasets in termsof challenging conditions and metadata. To addresssuch shortcomings, the C hallenging U nreal and R eal E nvironments for T raffic S ign D etection ( CURE-TSD )dataset was recently introduced [1], which was alsoutilized for traffic sign recognition in [2].The
CURE-TSD dataset was used to host the firstedition of the Video and Image Processing (VIP) Cup in2017 denoted as
Traffic Sign Detection under Challeng-ing Conditions . The VIP Cup is a student competitionin which undergraduate students form teams to workon real-life challenges. Each team should include one faculty member as an advisor, at most one graduatestudent as a mentor, and at least three but no more thanten undergraduate students. Formed teams participate inan open competition and the top three teams are selectedto present their work at the final competition, whichwas held at the 2017
IEEE International Conference onImage Processing (ICIP) in Beijing, China. Travel costsof finalist teams were supported by the IEEE SignalProcessing Society (SPS).In this article, we share an overview of the VIP Cupexperience including competition setup, teams, techni-cal approaches, statistics, and competition experiencethrough finalist teams members’ and organizers’ eyes.
Traffic Sign Recognition under Challenging Con-ditions:
Traffic signs can be recognized by state-of-the-art algorithms with high precision and accuracyin existing datasets, which are limited in terms ofchallenging conditions and corresponding metadata. Thelimited nature of these test sets makes it difficult toestimate the performance of recognition algorithms innon-ideal real-world scenarios. Recent studies [3, 4]showed that adversarial perturbations can degrade theperformance of existing traffic sign recognition systemsunder specific conditions. Even though these studiesshed a light on conditions that are intentionally designedto fool existing systems, introduced non-idealities areinherently different from realistic challenging condi-tions. To perform practical robustness tests for traffic
DRAFT (a) Darkness (b) Brightness (c) Snow(d) Haze (e) Rain (f) Dirty lens(g) Noise (h) Decolorization (i) Blur
Fig. 1: Challenging scene examples from the 2017 VIP cup.sign detection and recognition systems, we need to testthem with realistic mild-to-severe challenges, which isthe main objective of the VIP Cup 2017. The challengesin the VIP Cup include multiple levels of rain, snow,haze, brightness, darkness, shadow, blur, decolorization,codec error, dirty lens, and noise, whose examples aredepicted in Fig. 1.
VIP Cup 2017 Statistics:
The VIP Cup 2017 startedwith a global engagement of more than 250 requestsfrom 147 parties to access competition data from allaround the world as shown in Fig. 2. At the start line, the highest engagement was received from Bangladesh,India, China, and USA. At the registration stage, therewere 80 members clustered into 19 teams from 10countries including Australia, Bangladesh (2 teams),China (7 teams), Hong Kong (2 teams), India, Malta,Pakistan, Sweden, Taiwan, and USA (2 teams). Out ofthese 19 teams, 6 teams with a total of 32 members fromAustralia, Bangladesh (2 teams), China, Hong Kong,Sweeden, and Taiwan made it to the final stage.
Tasks in the VIP Cup 2017:
The VIP Cup 2017included an open competition stage and a final stage.
DRAFT
Australia: 2.7% Canada: 2.7% Taiwan: 2.7% Russia: 2.0% France: 1.4% Italy: 1.4% Sweden: 1.4% Turkey: 1.4%Belgium: 0.7% Colombia: 0.7% Egypt : 0.7% Germany : 0.7% Iceland : 0.7% Japan : 0.7% Lebanon : 0.7% Palestine : 0.7%Peru: 0.7% Saudi Arabia : 0.7% Singapore : 0.7% South Korea : 0.7% Spain : 0.7% UK : 0.7% Vietnam : 0.7% Bangladesh: 17.7% India: 15.6% China: 12.9% USA: 10.2% Brazil: 4.8% Pakistan: 4.8% Hong Kong: 4.1% Malta: 4.1%
Fig. 2: Global engagement map of the VIP Cup 2017.The call for competition was announced on February15, 2017 and the open competition started by makingdata publicly available on March 15, 2017. The videodataset was released in the open competition stage,which included processed versions of captured andsynthesized traffic videos with challenging conditionsspanning a wide range from mild to severe. The com-petition dataset was split into training set and test set. Specifically, , sequences were providedfor model development and , sequences for finaltesting. There were frames in each video sequence.Traffic signs within the video sequences included bi-cycle, goods vehicles, hump, no entry, no left, noovertaking, no parking, no right, no stopping, parking,priority to, speed limit, stop, and yield . The participantswere asked to develop algorithms that can detect thesetraffic signs under challenging conditions in the testset, which cannot be used in the model development.Participants were allowed to use Maltab ® as a coding platform and Python and C++ as coding languages alongwith any library or toolboxes. Competition rules, whichwere set to have a fair competition ground, guaranteereproducible research, and obtain practical algorithms,are as follows: • Any algorithm that utilizes future frames for pre-diction will be disqualified. • Any algorithm that utilizes testing labels in thefinal evaluation will be disqualified. • Any algorithm that utilizes testing sequences orlabels in the training including model training andvalidation will be disqualified. • The submissions should include detailed instruc-tions and necessary codes to replicate the results.Otherwise, the participants can be disqualified. • Reproducing results including training, testing, orany other processes should not exceed a reasonableamount of time that allows the organizers to evalu-ate all submissions within the given time window.
DRAFT
The open competition stage was completed on July8, 2017, which was the deadline to receive team sub-missions that included: • a report in the form of an IEEE conference paperup to six pages, on the technical details of themethods used, programs developed, and results; • estimated detection files for each test sequence; and • all codes with detailed comments and README files.The VIP Cup 2017 organizers evaluated the submis-sions and announced the finalists as team
Neurons ,team
PolyUTS , and team
Markovians on August 15,2017. Evaluation was based on overall precision, recall,and combination of these metrics. Finalist teams wereinvited to the final competition at the 2017
IEEE Inter-national Conference on Image Processing , which washeld in Beijing, China, September 17-20, 2017. Finalistteams presented their work on Sunday, September 17,2017. Each team had 15 minutes for their presentationand 5 minutes for questions and answers. After teampresentations, the jury had an internal discussion tofinalize the ranking. In the opening ceremony of theconference on September 18, 2017, IEEE SPS presidentDr. Rabab Ward highlighted the first Video and ImageProcessing Cup and publicly announced the winners ofthe competition. The jury included Dr. Amy Reibman,Dr. Batrice Pesquet, Dr. Patrizio Campisi, and Dr.Ghassan AlRegib.
Highlights of technical approaches:
All of the fi-nalist algorithms are based on state-of-the-art data-driven methods. Specifically, baseline methods used byfinalist algorithms rely on Convolutional Neural Net-works (CNNs) that directly learn visual representationsfrom examples with labels in a supervised fashion. Inthe VIP Cup, CNNs were utilized for various tasksincluding challenge type classification, preprocessing,localization, and recognition. The contribution of fi-nalist algorithms to the literature can be considered as four folds. First, challenging conditions in videosequences were identified by team
Markovians andteam
Nuerons . Second, video sequences were pro-cessed with challenge-specific operations to enhancetraffic sign visibility. Third, team
Markovians trackedsigns through temporal information, which is over-looked by majority of the state-of-the-art architecturesin the literature. Forth, a challenge-aware trackingmechanism was used by team
Markovians , whichalternated tracking mechanism based on the dominantchallenging condition in video sequences. The bestperforming method achieved a precision of . anda recall of . in the overall test set, which is anindicator of the competition difficulty and a sign ofroom for improvement. To achieve top performance,finalist teams used open-source deep learning librariesthat were commonly supported with GPUs. Participants’ opinions:
The VIP Cup 2017 createdan opportunity for students to work on a real-worldproblem related to disruptive autonomous vehicle tech-nologies. Even though participants had limited expe-rience, time, and resources, they delivered promis-ing algorithms, had successful presentations, and mostimportantly, showed unceasing dedication during thecompetition. Bangladesh University of Engineering andTechnology had a finalist team or honorable mentionin every SP Cup organized since 2014. Impressivelybut not surprisingly, two of the finalists are from theBangladesh University of Engineering and Technology.In this section, we share the inspirational stories andopinions of finalist teams about the 2017 VIP Cupexperience.
Team Neurons
Team
Neurons was formed by third year under-graduate students in the Department of Electrical andElectronic Engineering at Bangladesh University of En-gineering and Technology. Supervisor of team Neurons
DRAFT (a) (b)(c) (d)
Fig. 3: First team:
Neurons (a) with IEEE SPS president Dr. Rabab Ward, IEEE student services director Dr.Patrizio Campisi and VIP Cup 2017 organizer Dr. Ghassan AlRegib, (b) with judge Dr. B´eatrice Pesquet-Popescu,(c) with judge Dr. Amy R. Reibman, and (d) behind the scenes.
Affiliation:
Bangladesh University of Engineering and Technology
Undergraduate students:
Uday Kamal, Sowmitra Das, Abid Abrar
Supervisor:
Md Kamrul Hassan
Technical Approach:
Team
Neurons developed a data-driven system based on ConvolutionalNeural Networks (CNNs) similar to an existing approach [5] to identify the type of challengingconditions in a scene. Based on the identified challenging condition, they performed a prepro-cessing operation over video frames to eliminate the effect of challenging conditions and enhancetraffic sign visibility. After the preprocessing stage, they trained sepate CNNs to localize andrecognize traffic signs. In their algorithm development, they used the Keras API with Tensoflowback-end on NVIDIA GeForce GTX 1050 GPU.
DRAFT (a) (b)(c) (d)
Fig. 4: First runner-up:
PolyUTS (a) with IEEE SPS president Dr. Rabab Ward, IEEE student services directorDr. Patrizio Campisi and VIP Cup 2017 organizer Dr. Ghassan AlRegib, (b) VIP Cup 2017 final, (c-d) behind-the-scenes.
Affiliation:
University of Technology Sydney, Hong Kong Polytechnic University, University ofNew South Wales
Undergraduate students:
Weixi Feng, Aung Min, Jiawei Zhang, Chenhang He, Hardy Zhu,Wenqi Jia
Supervisor:
Xiangjian He
Technical Approach:
Team
PolyUTS utilized two Convolutional Neural Networks (CNN)trained with the competition dataset, one is for possible region proposal, and the other is forclassification. Region proposal network was based on extracting image features with a pretrainedCNN [6] and regressing these features. To reduce the time consumed during the region proposalprocedure, they forward propagated all possible bounding box coordinates once and used thesecond CNN architecture to classify proposed regions. In their algorithm development, theyutilized Tensorflow and OpenCV on NVIDIA GTX 1080 GPU.
DRAFT (a) (b)(c) (d)
Fig. 5: Second runner-up:
Markovians (a) with IEEE SPS president Dr. Rabab Ward, IEEE student servicesdirector Dr. Patrizio Campisi and VIP Cup 2017 organizer Dr. Ghassan AlRegib, (b-d) behind-the-scenes.
Affiliation:
Bangladesh University of Engineering and Technology
Undergraduate students:
Ahmed Maksud, Jubaer Hossain, Kinjol Barua, Roknuzzaman Rokon,Muhammad Suhail Najeeb, Nahian Ibn Hasan, Shahruk Hossain, Shakib Zaman, SM RaiyanChowdhury
Graduate mentor:
Sayeed Shafayet Chowdhury
Supervisor:
Mohammad Ariful Haque
Technical Approach:
Team
Markovians trained a recurrent Convolutional Neural Network(CNN) [7] to identify challenging conditions in a scene and detected traffic signs with a fasterregion-based CNN architecture. They used a Kalman-based approach to track signs with staticchallenging conditions including dirty lens and shadow whereas they used a Lucas-Kanade-basedapproach for all other challenge types. They used a CNN architecture to recognize detected andtracked signs. In their algorithm development, they used Keras API with Tensorflow back-endand OpenCV.
DRAFT professor Md. Kamrul Hasan was also the supervisor ofa finalist team in SP Cup, which is a great example ofsuccess when mentorship is combined with dedicatedstudents. Experience of team
Neurons can best bedescribed by their own words as follows: p “It was quite exciting when we heard the newsof IEEE VIP Cup being organized this year. Wedecided to participate as we thought that it wouldbe an excellent opportunity to test our new skill-setand gain some valuable experience doing real-world research... We had quite a lot of hardwarelimitations... Nevertheless, we went on with oursimulations in spite of the difficulties. Before ourRamadan vacation started, we got access to a PCwith a decent CPU and GPU configuration... Afterthat, there was no holding us back. We workedround the clock, often 15-16 hours a day, for 3whole weeks. A few days before submission, wedecided to add another network to our pipeline assome of the test results were not satisfactory. Itwas a hectic time. Even after submission, the battlewasnt over. We spent hours on end, in the middle ofthe night, exchanging emails with our reviewer andproviding clarifications for different parts of oursubmission package. We had to do all of this in themidst of our term-final, handling a lot of academicpressure at the same time. So, it was a great feelingof triumph, when we were selected as one of thefinalists and finally, the Champions of the IEEEVIP Cup 2017 - a feat none of us thought we couldachieve when we started off in this journey. It wasone of the happiest moments of our lives.” - Team Neurons p “ The first edition of IEEE VIP CUP was full of ’firstever’ experiences for me - first ever participation inany global competition, first ever research project,first ever international conference and so on. Participating in this competition helped me to learna lot not only about video and image processing,but also about machine learning. The challengeitself was very complex. To process this hugeamount of given dataset was also a tremendouschallenge for us. But above all, it was definitelya very exciting and rewarding experience! ”- UdayKamal p “ Participating in the VIP Cup was a very enrichingexperience. I learned about cutting-edge imageprocessing and machine-learning techniques likeneural networks; I learned how to read researchpapers and write one, give a technical presentation,collaborate with fellow team-members, and, stayfocused and motivated even in adverse situationsall of which are extremely valuable for doing anykind of research work. Besides, this is the first timeI attended an international conference, where I metresearchers and industry representatives leadingthe field of image-processing. But, most of all, Igot the opportunity to represent my country at aglobal stage, and compete on par with studentsfrom around the world. This really gives you asense of confidence that, if we are willing to put inthe effort, all of us could achieve things we wouldnteven dare to think of. ” - Sowmitra Das p “ It was really a unique experience for me. ICIP2017 was the first international conference that Ivejoined, and VIP Cup 2017 was the first internationalcompetition Ive participated in my undergraduatelife. Ive also learned a lot of new things throughoutthis journey. ” - Abid Abrar p “ The problem was interesting and quite challengingat least for the students of undergraduate level.I was hesitating in the beginning because of theteam members level in the UG program.....limited
DRAFT hardware resources in the lab for machinelearning....but the team relieved me within a fewweeks. It was enjoyable to see the spirit of theteam. ” - Kamrul Hasan, faculty mentor.
Team PolyUTS
PolyUTS is an international project team across theocean formed by junior and senior undergraduate stu-dents from the Hong Kong Polytechnic University andthe University of Technology Sydney. Team membersexpressed their opinions about the VIP Cup experienceas follows: p “ The collaboration between us broke the limitationof time and geology. It was through different socialplatforms that we consistently communicated witheach other. Finally, we were truly delighted thatour teamwork was such a great success. ” - Team
PolyUTS p “ This competition not just taught me a lot oftechnical related stuff but also let me experiencein how these competitions are conducted. Thatexperience is something I will not forget andremember for the rest of my life. ” - Aung Min p “ Detecting traffic signs in the given conditionsis really a tough task. Although I had learnedsomething about object detection before, I found itwas still hard to solve the problem at the beginning.The competition taught me that standing on theshoulder of giants was always a good starting point.By testing and evaluating algorithms proposed inothers papers, I gradually knew which methodswork for this task. Investigating why and how theywork finally instructed me what I should do. I thinkknowing how to overcome such a difficult problemis the most valuable reward for me. I do appreciatethe opportunity provided by the match. ” - Jiawei Zhang p “ Participating in VIP Cup 2017 was a greatexperience for me to learn more about machinelearning and how it is implemented. Although itwas a really challenging task, we still managedto complete a whole system. There were lots ofstruggles and confusion during the process, butcoming over them was also delightful. I’m veryhappy to spend my summer time on this competition. ” - Weixi Feng
Team Markovians
Team
Markovians was formed by junior undergradu-ate students from Bangladesh University of Engineeringand Technology. Common interest in video and imageprocessing as well as automation formed strong boundsbetween team members. As electrical engineering stu-dents and technology enthusiasts, team members werethrilled to contribute towards the research and develop-ment of a traffic sign detection system, which might beable to aid autonomous vehicles. Team
Markovians briefly expressed their opinions about their VIP Cupexperience as follows: p “ Although we had no prior expertise in the field,we did not stop short. We started out from zero andconstantly found ourselves at an impasse. However,the support from our supervisor and mentor andthe relentless efforts from all the team memberswas animating. We spent countless hours, groupsessions in weekends, through holidays and evenbefore exams for the project. The huge amountof data we needed to process required powerfulhardware which wasnt available to us. However,our perseverance allowed us to proceed with whatwe had and at the end we came up with a workingsolution. ” - Team
Markovians
DRAFT0
Organizers’ opinions:
It has been very inspiring tomeet the teams in person and hear their stories. Theirexperience demonstrated that with dedication one cancompete at the global level despite hardship, lack of re-sources, and limited support. In addition, this experienceshowed the importance of mentorship exerted here byprofessors who lead these undergraduate teams to excel.The dedication of competing teams motivated the orga-nizers to work tirelessly throughout the competition.
Upcoming VIP Cup:
The second edition of theVIP Cup will be held at the 2018
IEEE Interna-tional Conference on Image Processing in Athens,Greece, between October 7-10, 2018. The themeof the 2018 competition will be announced inFebruary. Teams who are interested in the VIPCup can visit: https://signalprocessingsociety.org/get-involved/video-image-processing-cup.In addition to the VIP Cup, the IEEE SignalProcessing Society announced the fifth edition ofthe SP Cup. The final competition will be heldat the IEEE International Conference on Acoustics,Speech and Singal Processing 2018 in Calgary, Al-berta, Canada, 15-20 April 2018. The theme of thecompetition is “Forensic Camera Model Identification”.For details, visit: https://signalprocessingsociety.org/get-involved/signal-processing-cup.
Acknowledgements:
The organizers of the VIP Cup2017 would like to acknowledge and express theirgratitude to everyone involved in the first VIP Cupjourney including but not limited to Patrizio Campisifor initiating the VIP Cup and his unmatched supportthroughout the competition; Min-Hung Chen and TariqAlshawi for their significant roles in the organizationincluding dataset preparation and algorithm validation;both Amy Reibman and B´eatrice Pesquet-Popescu forserving on the jury; and participating teams for theirhard work, dedication, and inspirational stories. Finally,the sponsorship by the MMSP Technical Committee of this Cup is much appreciated.
AuthorsDogancan Temel ([email protected]) isa postdoctoral fellow at the Georgia Institute ofTechnology. He was the recipient of the Best DoctoralDissertation Award from Sigma Xi honor society, theGraduate Research Assistant Excellent Award from theSchool of ECE, and the Outstanding Research Awardfrom the Center for Signal and Information Processingat Georgia Tech.
Ghassan AlRegib ([email protected]) is currentlya Professor in the School of Electrical and ComputerEngineering at the Georgia Institute of Technology.He is the director of Omni Lab for Intelligent VisualEngineering and Science (OLIVES) and the Center forEnergy and Geo Processing (CeGP). He is a Memberof the IEEE SPS MMSP and IVMSP TechnicalCommittees. He served at various capacities within theIEEE SPS Society including the Technical Programco-Chair of GlobalSIP 2014, Tutorial Chair for ICIP16,and Technical Program Chair of ICIP 2020. He receivedvarious awards including the Outstanding Junior FacultyAward in 2008 and the Global Engagement ExcellenceAward in 2017. R
EFERENCES [1] “CURE-TSD: Challenging unreal and real en-vironments for traffic sign detection,” https://ghassanalregib.com/cure-tsd/.[2] D. Temel, G. Kwon, M. Prabhushankar, and G. Al-Regib, “CURE-TSR: Challenging unreal and realenvironments for traffic sign recognition,” in
NIPS,MLITS Workshop , 2017.[3] J. Lu, H. Sibai, E. Fabry, and D. Forsyth, “Noneed to worry about adversarial examples in objectdetection in autonomous vehicles,” in
IEEE CVPR,Spotlight Oral Workshop , 2017.
DRAFT1 [4] N. Das et al. , “Keeping the bad guys out: Protectingand vaccinating deep learning with jpeg compres-sion,” in arXiv:1705.02900 , 2017.[5] K. Simonyan and A. Zisserman, “Very deep con-volutional networks for large-scale image recogni-tion,” in
ICLR , 2015.[6] C. Szegedy et al. , “Going deeper with convolu-tions,” in
IEEE CVPR , 2015.[7] M. Liang and X. Hu, “Recurrent convolutionalneural network for object recognition,” in
Proc.IEEE CVPR , June 2015, pp. 3367–3375., June 2015, pp. 3367–3375.