Recent experimental techniques allowed study of the relationship between neurons stimulus-preference and connectivity. In particular, in the layer II/III of primary visual cortex, it was shown that excitatory neurons with the same orientation preference have a high probability of being bidirectionally connected. However, the intracortical connectivity is only getting refined after eye-opening. We have recently hypothesized that this process is a result of experience-dependent plasticity, modelled by a Hebbian learning. In contrast to excitatory neurons, parvalbumin-expressing (PV) inhibitory cells are less input-specific. PV neurons receive excitatory inputs from neurons with different orientation preferences. In this work, we investigate the mechanism by which excitatory to inhibitory connections are formed (how) and their potential function (why) in a small recurrent network. We found that a model combining Hebbian learning with homeostatic plasticity, which allows PV neurons to spike at a high rate (i.e reproducing the fast-spiking intrinsic property of the cells), develops unspecific excitatory-to-inhibitory connections (Figure 1). We then tested the role of inhibition by simulating our model with and without inhibition after learning convergence. We found that inhibition ensures less fluctuation of the synaptic weights over time, hence stabilizes the network. We therefore propose that unspecific excitatory to PV connections can be a result of the intrinsic homeostatic property of PV neurons, and can allow the network to be more stable.