The cutout relay opens the path between the armature and the battery.
This relay closes when the generator RPM is high enough to generate a
voltage that is equal or exceeds the battery voltage. When the cutout
relay closes the heavy wire coil on the relay adds more flux to keep
the relay closed. When the generator RPM decreases so that the
generator voltage is less than the battery voltage, the battery tries
to "motor" the generator and the current through the heavy wire
winding reverses and opposes the flux from the fine wire winding
causing the relay to open which disconnects the generator from the
battery.

This sounds like the most reasonable thing happening. I'm thinking that a fine
adjustment on the cutout relay tension might be all that is needed.

Jim