Practical Course: Vision-based Navigation IN2106 (6h SWS / 10 ECTS)
WS 2020, TU München
Lecturers: Nikolaus Demmel, David Schubert
Please direct questions to visnav_ws2020@vision.in.tum.de
News
- 2020-12-07: We'll present project topic suggestions in today's tutor session. (Update: project slides are now linked below)
- 2020-11-09: You can now sign up for Gitlab credentials here.
- 2020-11-09: Course information updated. Lectures are now available. Join the first tutor session today at 4pm. Watch the first lecture video by then. Gitlab accounts are still pending (more info via email).
- 2020-07-17: The start date of the course and the presentation date may change due to the apparent later start of the lecture period this winter (02.11.?). We want to start with the first tutor session on the Monday of the second week of the lecture period. We'll wait to see how things develop.
- 2020-07-08: Added pre-meeting slides and recording.
- 2020-07-07: We updated below course details. The deadline for registration through the matching system and for sending an email about prerequisites is 21.07.2020 (see below "Pre-meeting and Registration").
- 2020-06-22:
Some of the info below is still from SS20. Course details will be updated in the following days.We will be using the IN.TUM-Matching-System. The pre-meeting will be held online some time in the first two weeks of July. Check back here in the next few days.
COVID-19
Update: The course is fully online.
Time & Date
- Lecture & exercises (assignment phase): Mondays. Watch video lectures 2pm to 4pm (or beforehand in your own time). Virtual tutor sessions 4pm to 6pm on BBB.
- Individual weekly meetings (project phase): Fixed 30min time-slot for each project/group, preferably Mondays between 2pm and 6pm. Arranged after assignment phase.
- Project presentations: 08.02.2020, 2pm (tentative)
- Project report due: 15.03.2020 (tentative)
- Maybe: Use of computer labs for exercise and project work.
Feedback and Suggestions
This is the second time we conduct this course online. We are also learning in the process. If you have concrete suggestions on how to improve the experience for students, please let us know at visnav_ws2020@vision.in.tum.de, and we will consider them for this or the future iterations.
During the semester there will be an anonymous lecture evaluation. We encourage all participants to fill out this short survey. The individual link will be sent through TUMonline.
The results of anonymous "lecture evaluation" can be found here. Thanks for your feedback!
Resources and Platforms
- All material related to the course can be found on this page. (We don't use Moodle.)
- We distribute source code and manage submission of exercises through our chair's Gitlab. You are encouraged to ask technical questions on the common issue tracker. Please also search for existing open and closed issues and feel free to answer other student's questions if you can. You can sign up for Gitlab credentials here.
- Video lectures should be watched weekly.
- Programming assignments should be started every week and are to be submitted after two weeks (you should aim to finish each sheet within 1 week, so you can start the next one).
- Virtual Tutor Sessions are held weekly with Q&A on lecture and exercises and individual help on programming assignments.
- Student's are encouraged to interact on the course's student matrix chat room.
- The course instructors can be reached via visnav_ws2020@vision.in.tum.de.
- Non-public material and links are found here.
Course Structure
In the initial phase (approx. 5 weeks), there will be lectures to refresh some relevant background material and prepare you for the programming assignments. A new programming assignment sheets should be started and discussed every week. They need to be solved individually, but students are encouraged to discuss while working on the solutions.
In a second phase, you will work in teams of 1-2 students on a practical problem (project). For the rest of the semester, the group meets weekly with their tutors (in a video chat) and presents and discusses their progress. At the end of the course, the teams will present their project in a talk and demonstrate their solutions. They will document their project work in a written report. Both the assignments and the project part will be graded, and together determine the final grade.
Grading
The final grade will be determined by both the programming assignments and the project.
For grading the programming assignments we consider completeness and timely submission (not so much things such as code quality, as long as it works). Note that you have to complete all exercise sheets to pass the course, even if you miss a submission deadline.
In the project phase the main focus is on your implementation, on the presentation and the report, but we will also consider how you approach the problem, how you engage with your tutors and how you manage your time.
Lectures
This practical course assumes some prior knowledge of relevant theory and we don't have the time to revisit all pre-requisites. However, during the initial assignment phase of the practical course we refresh some relevant background material in form of weekly video lectures, that should prepare you for the programming assignments. You have the opportunity to ask questions during the weekly Q&A session on Monday 4-6pm. If you haven't watched the lecture before, please do so during the lecture time-slot Monday 2-4pm, to be prepared for the tutor session.
Videos and Slides
Slides and recordings for the lectures can be found here (recorded summer 2020).
Discussed on | Video | Slides | Content |
---|---|---|---|
08.07.2020 | premeeting.pdf | Overview (Pre-Meeting) | |
09.11.2020 | Lecture 1 (YouTube) | visnav_lecture1.pdf | Introduction, 3D Geometry, Lie Groups |
16.11.2020 | Lecture 2 (YouTube) | visnav_lecture2.pdf | Camera Models, Optimization Theory |
23.11.2020 | Lecture 3 (YouTube) | visnav_lecture3.pdf | Keypoint Detection, Matching, Motion Estimation |
30.11.2020 | Lecture 4 (YouTube) | visnav_lecture4.pdf | SfM / SLAM, Bundle Adjustment, Triangulation |
07.12.2020 | tutor session recording | projects.pdf | Project Phase Modalities and Topics |
Exercise Sheets
For the first 5 weeks, we distribute weekly exercise sheets that you should work on. You are encouraged to interact with your fellow students and share ideas, but these mandatory exercises have to be solved and handed in by every student individually.
The exercise sheets are distributed together with the code framework via Gitlab. Hand-in of exercises is also done in Gitlab via merge-request. You should aim to finish each sheet within a week, but the deadline for submission for each sheet is 2 weeks after it was initially discussed. The exact deadline is noted on each exercise sheet. This gives you a second chance for each sheet to ask remaining questions in the tutor session. We strongly encourage students to not fall behind the weekly schedule, in particular in the first weeks, as some of the later exercises may require more work.
Virtual Tutor Sessions
We hold virtual tutor sessions on Monday 4-6pm, starting November 9th. For this we are using the Informatics department's BigBlueButton server. This offers video-conferencing from your browser. Details on how to join the virtual tutor sessions can be found here.
We will start with general announcements, then do a common Q&A session for the latest lecture as well as the current and the previous exercise sheet. The usefulness for everyone depends on you asking questions. We assume you have already watched the latest lecture at this point. Please consider taking notes while watching the lecture to prepare questions for the Q&A session.
Afterwards, you are encouraged to work on the exercises. The call will remain open and you can speak to us about any issues that come up. While you can also work on the exercises in your own time, we encourage you to make use of the tutor sessions as much as possible, as most questions – that might hold you up otherwise – can usually be resolved quickly during the tutor session.
Please use your real name when joining the call. Attendance is voluntary, but highly encouraged for successful participation. Since some of you might not be able to join due to time-zone or other extraordinary circumstances, the sessions will be recorded.
In case you cannot attend, you may also send us questions (on the latest lecture as well as the current and the previous exercise sheet) to visnav_ws2020@vision.in.tum.de and we will try to address them in the Q&A session. Please send us the questions at least one day in advance, if possible.
Projects
After the lecture and assignment phase is completed, students work in groups of 1-2 people on a more open-ended project. We will present some example projects, but you may also suggest your own.
See the projects slides (TBA) for more details.
Inter-Student Communication
Our experience has shown that students significantly benefit from discussing and talking to their fellow students. Since without physically attending lectures and exercises it is difficult to meet fellow students, we have set up a student chatroom for discussions related to this course. Please note that this is not an official communication channel to reach the course organisers (please use email for this), but rather to provide a common space where students taking this course can meet each other and discuss things related to the course. Details on how to join the student chat room can be found here.
For technical discussions and issues about the exercises, you are also invited to interact on the Gitlab issue tracker.
During the first virtual tutor session we will furthermore offer a rounds of introductions, where you may present yourself to the group (1 minute max per student). This is voluntary.
Pre-meeting and Registration
Places are assigned through TUM matching system. Please see http://docmatching.in.tum.de/ for the general procedure and for important dates (matching registration deadline is 21.07.2020).
TUMOnline course entry: https://campus.tum.de/tumonline/wbLv.wbShowLVDetail?pStpSpNr=950526022
Pre-meeting for more information about the course content and procedure will be held online on 08.07.2020 at 1:pm. Attendance to the pre-meeting is not required for participation in the course, but registration through the matching system is. You can find details about the registration procedure in the pre-meeting slides or video recording.
You are required to send information about your prior experience to verify prerequisites before the end of the matching deadline (21.07.2020). Please consult the pre-meeting slides for instructions on what information to send.
Prerequisites
To participate in the course you need to fulfil the following requirements:
- Good knowledge of the C/C++ language is essential
- Good knowledge of basic mathematics such as linear algebra, calculus, and numerics is required
- Participation in at least one of the following lectures of the TUM Computer Vision Group:
- Computer Vision I: Variational Methods
- Computer Vision II: Multiple View Geometry
- Similar lectures can also be accepted, please contact us.
Number of participants: max. 12
Course Description
Vision-based localization, mapping, and navigation has recently seen tremendous progress in computer vision and robotics research. Such methods already have a strong impact on applications in fields such as robotics and augmented reality.
In this course, students will develop and implement algorithms for visual navigation and 3D-reconstruction, relevant for applications such as autonomous navigation of wheeled robots and quadrocopters, tracking of handheld devices, or 3D reconstruction. The investigated algorithms may include, visual odometry, structure from motion, simultaneous localization and mapping with monocular, stereo, or RGB-D cameras, (semi-)dense 3D reconstruction.
Course Layout
- Lecture & Exercise : up to 2 hours per week recorded video lecture; 2 hours per week tutored Q&A and exercise session, Mondays from 4pm to 6pm. There are 5 lecture & exercise sessions. Each week, the exercise for the following week will be announced and has do be handed in online by each student individually within 2 weeks. Attendance to lecture and tutor session is voluntary, but highly encouraged.
- Project : After the initial 5 weeks, student should form groups of 1-2. Each group will be assigned to a project. Students can work on their own and consult the tutors in a weekly meeting to discuss project progress and next steps. Attendance to meetings with tutors is mandatory.
- Presentation and demo : Each group will be assigned a time slot on one of the last days of the semester to present their results, followed by a Q&A session. The presentation shall be 12 minutes long + 3 minutes for questions. The presentation should comprise 5-10 slides to explain the project goals and results in to fellow students and may include a short live demo or video.
- Project Report : Each group writes a report on their project work (10-12 pages, single column, single-spaced lines, 11pt font size; title page, table of content and references will not be accounted for in the page numbers). The report should summarize the project goals, what was implemented, and what results were obtained.
Literature
A good introduction to many aspects of computer vision relevant for the practical project is the following course, which has recordings on YouTube:
- Computer Vision II: Multiple View Geometry, https://vision.in.tum.de/teaching/online/mvg
The following book also covers many aspects. You should focus on Part II and III and selected background from Part I as needed:
- Timothy D. Barfoot, "State Estimation for Robotics", July 2017, Cambridge University Press
- Free pdf available: http://asrl.utias.utoronto.ca/~tdb/bib/barfoot_ser17.pdf
Less relevant, but still helpful:
- Autonomous Navigation for Flying Robots (Online lectures and EdX course), https://vision.in.tum.de/teaching/online/visnavfly
- Computer Vision I: Variational Methods, https://vision.in.tum.de/teaching/online/cvvm
Selected publications:
- Edward Rosten et al., "Faster and better: a machine learning approach to corner detection" (https://arxiv.org/pdf/0810.2434.pdf)
- Michael Calonder et al., "BRIEF: Binary Robust Independent Elementary Features" (https://infoscience.epfl.ch/record/149242/files/top_1.pdf
- Ethan Rublee et al., "ORB: an efficient alternative to SIFT or SURF" (http://www.willowgarage.com/sites/default/files/orb_final.pdf)
- Raúl Mur-Artal et al., "ORB-SLAM: A Versatile and Accurate Monocular SLAM System" (http://webdiis.unizar.es/~raulmur/MurMontielTardosTRO15.pdf)
- Ethan Eade, "Lie Groups for 2D and 3D Transformations" (http://ethaneade.com/lie.pdf) –> compare also chapter 7 in the Barfoot book mentioned above