“You didn’t understand what he was saying. I didn’t, either.” means:

Either you or I understood what he was saying.
Just as you understood what he was saying so did I.
Neither you nor I understood what he was saying.
Both you and I understood what he was saying.
Table of English Grammar Contents (All Levels)
English Worksheets
Upper-Intermediate Grammar Exercises
English Vocabulary Tests