CSCI 252: Neural Networks (and Graphical Models)


General Information

Professor: Simon D. Levy
Lecture: MWF 10:10 - 11:05 AM
Parmly 405
Office: Parmly 407B
E-mail: simon.d.levy@gmail.com
  Office Hours: Daily 11:15-12:10, 2:30-3:25 and by appointment

Textbook: There is no textbook for this course. In my experience, neural-net textbooks are either (1) very expensive; (2) unnecessarily mathematical; (3) incomplete; or (4) some combination of the above. We will rely on primary source material – both classic and contemporary – available online.


Course Objectives

By the end of the course you will be able to

  1. Explain the history of neural network research and critique arguments for and against neural networks, both practical and theoretical.

  2. Implement the most popular neural-network algorithms in Python.

  3. Look at a computational problem and determine whether / which one of these approaches is appropriate for solving it.

We will take an “architecture-of-the-week” survey approach to the material, attempting to cover one kind of neural net each week.


Professionalism

At this point in your CS career you probably realize that what matters most is the quality of the work (software) you create and the impression you make as a serious student and software professional. Hence:

  1. If your code produces a syntax error or fatal runtime error in Python (or whatever environment we're using), you'll get a zero on that part of the assignment, as well as any other parts that depend on that program.

  2. No late work will be accepted, or make-up exams given, without a note from the Dean's office.
The best way to avoid the “But the program worked when I ran it!” problem is to submit your code early to sakai , download it from sakai into an empty folder, and run it yourself.


Grading

These percentages are flexible. If you have a bad day and don't do well on an exam or assignment, I will try and weight your other work more heavily.

The grading scale will be 93-100 A; 90-92 A-; 87-89 B+; 83-86 B; 80-82 B-; 77-79 C+; 73-76 C; 70-72 C-; 67-69 D+; 63-66 D; 60-62 D-; below 60 F.


Honor System

The exams will be done without books or notes and without assistance from other people. You can discuss your programming assignments with other students in the class, but you should not share code. Talk to me before asking another student for help!


Final Exam

The final exam for this course will be given during the final exam week. You can take this exam during any of the regularly scheduled exam periods that week. You must supply an exam envelope to the instructor or the department administrative assistant no later than noon on the last day of class. You must specify a provisional day and time on the envelope, which you are free to change on the clipboard provided outside the door of Parmly 407 any time that week. Email or phone requests to reschedule will not be accepted.

The exam will be given in Parmly 405, and you should arrive promptly before the appointed time. If you are more than 15 minutes late, you will have to reschedule your exam. If you are more than 15 minutes late to the last exam period on Friday afternoon, you will receive a grade of 0 on your exam.

Students who have approved academic accommodations must make arrangements to use those accommodations directly with the instructor no later than the last day of class. Students approved for extra time will receive that time at the tail end of the morning exam period or before the beginning of the afternoon exam period (for example, ending at 1:30 PM for a morning exam or beginning at 12:30 PM for an afternoon exam). Students approved for a low-distraction testing location should reserve that space during the last week of classes, following instructions distributed by Dean Price (sophomores, juniors or seniors) or Director of Disability Resources Lauren Kozak (first-years).


Accommodations

Washington and Lee University makes reasonable academic accommodations for qualified students with disabilities. All undergraduate accommodations must be approved through the Office of the Dean of the College. Students requesting accommodations for this course should present an official accommodation letter within the first two weeks of the (fall or winter) term and schedule a meeting outside of class time to discuss accommodations. It is the student's responsibility to present this paperwork in a timely fashion and to follow up about accommodation arrangements. Accommodations for test-taking should be arranged with the professor at least a week before the date of the test or exam.


Programming assignments

Programming assignments should be submitted through your dropbox on Sakai. Each assignment is due 11:59 PM on the day posted.

The computers in the CS Department should all have NumPy and matplotlib installed, but I encourage you install them your laptop or home computer and work on these assignments anywhere you like. Experience has taught me that the students who do best in this course are the ones who start the assignments early and take advantage of office hours for help.

  1. Assignment 1 (due Mon 19 Sep)

  2. Assignment 2 (due Friday 30 Sep)

  3. Assignment 3 (due Wednesday 12 Oct)

  4. Assignment 4 (due Monday 24 Oct)

  5. Assignment 5 (due Monday 07 Nov)

  6. Assignment 6 (due Friday 18 Nov)

  7. Assignment 7 (due Friday 09 Dec)



Tentative Schedule of Lectures and Exams

 

Monday

Wednesday

Friday



05 Sep
Week 0
    Course Overview;
Philosophical Foundations


12 Sep
Week 1
Reading: The Appeal of Parallel Distributed Processing (Chapter 1 of PDP Volume 1, 1986)

Lecture Notes

Dot Product Lecture Notes

cosine.py

eucdist.py

Dot Product Lecture Notes


19 Sep Week 2
Architecture #1: Kohonen's Self-Organizing Map

Lecture notes

Reading: Törmä 1994

Due:Assignment #1

Kohonen SOM, continued

Reading: Zhao, Li, Kohonen 2010

Lecture notes



26 Sep
Week 3
Architecture #2: The Hopfield Network

Lecture notes

Reading: Hopfield 1982

Reading: Hopfield & Tank 1985

Due:Assignment #2



03 Oct
Week 4
Architecture #3: Sparse Distributed Memory

Lecture notes

Reading: Denning 1989

Watch at home: Jeff Hawkins' TED Talk on SDM

Watch in class: Jeff Hawkins on Modeling Data Streams Using Sparse Distributed Representations (Right-click and open in new tab/window)



10 Oct.
Week 5
In-class Exam #1 Review Exam #1

Due:Assignment #3

Reading Day; No class


17 Oct
Week 6
Architecture # 4: Neuromorphic Vision Chips

Reading: Koch & Mathur (1996)

Lecture Notes

Guest lecture by Dr. Geof Barrows (Centeye, Inc.)

Reading: Barrows, Srinivasan, Chahl (2003), sent by email

Image Convolution

Lecture Notes



24 Oct
Week 7
Architecture #5: Latent Semantic Analysis

Lecture Notes

Reading: Landauer & Dumais (1997)

Optional: Davis (2006)

Due:Assignment #4

LSA, continued

Hands-on session: Image convolution and optical flow (Assignment #5)


31 Oct
Week 8
Architecture #6: Tensor Products

Reading: Smolensky 1990

Lecture Notes

Review for Exam #2 In-class Exam #2


07 Nov
Week 9

Lecture Notes

Reading: Gayler (2003)

Due:Assignment #5

Tensor Products, part b (continued)

Tensor Products, part b (continued)



14 Nov
Week 10
Guest lecture by John Donaldson '92 (Pandora) Reading: Kanerva 2008

Due:Assignment #6



28 Nov
Week 11
Architecture #7: The Simple Reucurrent Network

Reading: Elman (1990)

Lecture Notes

SRN, continued SRN, continued


05 Dec
Week 12
Architecture #8: Long Short-Term Memory

Reading: Understanding LSTM Networks

Lecture Notes

Reading: Zero-Shot Translation Review for final exam

Due:Assignment #7