Don't give up hope. Some years are just rubbish. Also those look like Tahitian limes (smooth skin, lots of juicy flesh) rather than Kaffir (very knobbly skin with little juice). Kaffirs are usually grown for their 2 lobed leaves rather than fruit.
Is it Lemon Scab? "Lemon Scab affects lemons and some mandarin varieties. It presents
as raised grey to brown corky scabs on the smaller branches, leaves and fruit, causing peaked distortions on the fruit. It may also cause fruit drop
at the embryo stage, when fruit is very small. Lemon scab is caused
by a fungus which attacks when the petals are falling from the flowers."
"Spray when about half the flower petals have dropped, using copper spray mixed with white oil. Take care to mix the white oil separately, and then add to the already prepared copper spray. Two tablespoons of white oil to one litre of water, and add this to 4.5 litres of prepared copper
spray." http://thegarden.com.au/2011/06/citrus-pests-diseases/ @kathyb