Oh no, not stupid question at all. But a lot of factors are involved.
For most efficient flight the plane would be accellerated as fast as possible so the flaps and slots can be cleaned up as soon as possible. When? it depends on weight - pax- and fuel load.
But very often that is rather irrelevant because the flight profile is dictated by ATC - getting out of the way for the next take-off. Or by passing over "noise sensitive areas". Sometimes a plane will climb fast at slow speed in the beginning, then continue slowly at reduced power over a noise sensitive area, then climb steeply again, continue slowly over another sensitive area, and then finally speed up and clean the wings. That's one of the reasons why flight deck crews must train on simms and read a lot of "books".
But the easy answer to your question is that at any given take off weight he will know an "indicated minimum airspeed" at which he will retract flaps. Indicated airspeed is at sea level identical to true airspeed, but at any higher altitude it is a higher true airspeed compensating exactly for the reduced reaction on the wing of the thinner air. The indicated airspeed can be read on the instrument panel. It's of course also a very important figure when programming the autopilot.
Hope that helped.
Best regards, Preben Norholm
Always keep your number of landings equal to your number of take-offs