No crank when car is cold

For the past year or so, my car would not crank on the first try but would on the 2nd or 3rd try this only happened once in a blue moon. But for the past 2-3 weeks it has to a point that it does not crank at all. My car is manual, so when that happens I push it and put it into gear which makes the car start. After driving the car when I turn it off and try to start it again, it does crank. Sometimes I would have to hold the ignition in the start position for 3-4 seconds before the engine cranks. I am trying to find out which part is causing this malfunction. Anyone had a similar experience that can point me to the right direction? Also I notices that my car also cranks even if I don't hold down the clutch. I don't know if the last owner bypassed the clutch switch or that's where the malfunction is coming from.