Work those fingers! Teachers I had always told me to work the scales, up and down the fingerboard. With a pick it was up stroke/downstroke. With fingers it was alternately i m or i a, free stroke, then rest stroke. Try out this site my nephew has recommended to me; he uses it everyday to build up finger dexterity and speed gain– guitarcardio.com
Note: guitar cardio is in tab form and better suited for electric guitarists using a pick. For classical guitarists who read music I recommend Segovia’s multi-position diatonic scales.