If you have a Realtek audio card then you must configure it to use the 2 speaker setup to record the system audio using Screencast-O-Matic.
1. Go to Control Panel
2. Open "Realtek HD Audio Manager"
3. Select "2 / 2.1 Speaker" for Speaker Configuration (as shown below)