setGregorianChange(new Date(Long.MIN_VALUE)) does not convert the calendar to pure GregorianCalendar

The calendar you're using is fine - you just need to remember that a Date only stores an instant in time. It doesn't remember a calendar, time zone or text format.

The Date value you've got is correct - it has an epoch millis of -62072462953058, which corresponds to 0003-01-01T17:10:46Z (at least for me - depends on time at which you run the code) in a system which doesn't use the Gregorian change.

However, Date.toString() formats that instant as if there was a Gregorian change, leading to the two day offset.

You can see this by using Java 8's Instant instead:

import java.time.*;
import java.util.*;

public class Test {
    public static void main (String args[]) {
        GregorianCalendar test = new GregorianCalendar();
        test.setGregorianChange(new Date(Long.MIN_VALUE));
        test.set(Calendar.DAY_OF_MONTH, 31);
        test.set(Calendar.MONTH, 11);
        test.set(Calendar.YEAR, 1600);
        test.add(Calendar.DAY_OF_MONTH,-583657);
        Date date = test.getTime();
        Instant instant = Instant.ofEpochMilli(date.getTime());
        System.out.println(date);
        System.out.println(instant);
    }
}

Output (at the moment, in London):

Wed Jan 03 17:15:26 GMT 3
0003-01-01T17:15:26.913Z

The two values represent the same point in time, but the Instant conversion to string doesn't perform a Julian/Gregorian cutover, whereas Date does.