Very likely that the existing clay soil is not 100% clay. It's most likely a clayey silt. Possibly even has some sand sized grains in there. Probably washing out of the backfill that doesn't have good grass cover just yet. It all washed off together (clay, silt, and sand) but the clay is fine enough to stay in suspension and go on across the sidewalk or down the driveway and to the gutter. As soon as the velocity of the water dropped just a bit, the heavier sand and silt dropped out and stayed behind.
Sounds like nothing to worry about. Just make sure there aren't any holes or washouts that need filling, and you should be good.