Health chiefs fear people from parts of Birmingham will end up in hospital due to a lack of uptake of flu jabs.