“You didn’t understand what he was saying. I didn’t, either.” means:

Just as you understood what he was saying so did I.
Either you or I understood what he was saying.
Neither you nor I understood what he was saying.
Both you and I understood what he was saying.
Upper-Intermediate Grammar Exercises
English Worksheets
Advanced Grammar Exercises
English Modals Tests