In this paper, we show that non-uniform distributions in swarms of agents have an impact on the scalability of collective decision-making. In particular, we highlight the relevance of noise-induced bistability in very sparse swarm systems and the failure of these systems to scale. Our work is based on three decision models. In the first model, each agent can change its decision after being recruited by a nearby agent. The second model captures the dynamics of dense swarms controlled by the majority rule (i.e., agents switch their opinion to comply with that of the majority of their neighbors). The third model combines the first two, with the aim of studying the role of non-uniform swarm density in the performance of collective decision-making. Based on the three models, we formulate a set of requirements for convergence and scalability in collective decision-making.