“You didn’t understand what he was saying. I didn’t, either.” means:

Just as you understood what he was saying so did I.
Either you or I understood what he was saying.
Neither you nor I understood what he was saying.
Both you and I understood what he was saying.
English Modals Tests
Elementary Grammar Exercises A1
Intermediate Grammar Exercises
English Vocabulary Tests