The preparation of controllable small noble metal nanoparticle (NP) aggregates is required for plasmon-enhanced spectroscopy and metamaterials development. In this work, we present a systematic study of three aromatic diamines as possible linkers of silver NPs. Symmetric 4,4′-derivatives of bibenzyl, stilbene and tolane have been selected to reveal the effect of linker rigidity and conjugation degree on the morphology and optical properties of the obtained aggregates. Surface-enhanced Raman scattering spectra of colloidal solutions with different aromatic amine additives have been acquired to assess the surface equilibria and adsorbate state. The non-monotonic concentration profiles have been obtained in the case of effective NPs linking and can be considered as its feature. Transmission electron microscopy and absorption spectroscopy have been involved for the analysis of size and morphology of NPs aggregates. Effective linking of silver NPs is established for stilbene and tolane compounds, which give quasi-spherical aggregates and elongated dimers/trimers. Surface-initiated azo conversion is revealed for bibenzyl compound instead of NPs bridging. The tendency to decrease the degree of NP aggregation with increasing aromaticity and “springiness” of molecular linker is observed. The capability of rigid diamines to assemble the nanoparticles with a finite degree of aggregation, even at high concentrations, distinguishes them advantageously from aliphatic bifunctional linkers. Highly stable plasmonic aggregates can be obtained in this way and implemented further as building blocks for nano- and meta-materials, as well as for tags in imaging applications.